July 26, 1996
Submitted to Journal of Computational Organization Theory
This paper is based on a version originally published in K. M. Carley and M. J. Prietula (Eds.), Computational Organization Theory. Hillsdale, NJ: Lawrence Erlbaum, 1994, pp. 19-38.
The author gratefully acknowledges the support of the Ameritech Foundation, through the Information and Organizations Program of the University of Michigan Institute for Public Policy Studies, the University of Michigan Center for Parallel Computing, which is partially funded under NSF Grant CDA-92-14296 and the Massachusetts Institute of Technology Center for Coordination Science and NSF IRI Grant 92-24093.
The paper has benefited from discussions with Michael Gordon, Helen Klein, Michael Cohen, Brian Pentland, Martha Feldman, Tom Finholt, Ojelanki Ngwenyama and Kathleen Carley.
Such rapid changes may pose problems for organizations that have evolved to fit the old environment. In particular, the changes may make novel organizational forms more appropriate. Organizational ecologists explain the diversity of forms by analogy to biological species competing for resources (e.g., Singh and Lumsden, 1990) those with forms more appropriate for the environment are more successful at acquiring resources and thus tend to survive while those with less effective forms tend to fail. Over time, this selection results in an observable match between organizational forms and the environment in which they operate.
Unfortunately, these theories do not provide much insight into exactly what kinds of new form might become desirable; they say only that if the environment changes, new forms may eventually replace existing ones and describe the processes by which this occurs. Indeed, Romanelli (1991) calls the question of the origin of new organizational forms, "one of the critical unaddressed issues in organizational sociology".
In this paper, I will discuss the use of computer simulations to search for novel organizational forms by reproducing some of the mechanics of organizational evolution. The metaphor of organizations competing and being selected on the basis of their fitness is a compelling one. Natural selection is also the basis of a heuristic search technique known as the genetic algorithm (GA) used to search large problem spaces (Holland, 1992.) If we can describe the space of organizational forms, we can use this algorithm to search it.
Using the GA to search for possible organizational forms has the advantage that variations in organizational forms explored are not restricted by social factors, institutional pressures (DiMaggio and Powell, 1983) or human ingenuity. However, this advantage is simultaneously a major concern about simulations: the potential lack of external validity. Computer models necessarily abstract from real organizations; the features simulated must be chosen carefully to ensure that conclusions drawn from the models can be applied more generally (Burton and Obel, 1980).
In the rest of this paper, I describe the GA and discuss some issues in applying it to organizational forms. To illustrate the potential of this approach, I then present a simple model of an organization and some preliminary results from using the GA. I conclude by discussing possible bases for a more general organizational model and general caveats about the approach.
While the GA seems random, it turns out to be an effective method for searching a large search space and it has been used to evolve solutions to many kinds of problems, including strategic games, optimization problems, mechanical design and image classification (Booker, et al., 1987) and even LISP programs (Koza, 1992). It can be shown that the individuals in the current population encode large amounts of useful information about combinations of features, called schemata (Holland, 1992). These schemata implicitly represent numerous similar individuals not actually present in the population. Furthermore, as the individuals are selected and bred, schemata are also reproduced in the population in proportion to their fitness. In essence, by manipulating a modestly sized population, a GA implicitly searches in parallel a much larger portion of the space.
As the above discussion make clear, there are two main issues in applying the GA to a problem such as organizational design.
Representation. First, we need a representation of the organizations to be evolved, general enough to represent any organizational form of interest. Representing only different kinds of hierarchies, for example, would not allow us to consider market mechanisms for the same transactions (Williamson, 1975).
Fitness. Second, we need to represent the environment in which the organizations perform and, explicitly or implicitly, a fitness function to identify "good" organizations. By varying the environment, we can look for forms that may be desirable under different conditions.
The use of the GA does place some constraints on the representation, however. First, we require a representation in which cross-overs between individuals make sense. This requirement implies viewing an organization as a collection of features which can be combined and recombined in various ways--what Romanelli (1991) and Hannan and Freeman (1986) describe as organizational genetics. Most models of organizations do not have this property. For example, Hannan and Freeman (1986) note that much of organizational research has used conventional classifications of types of organizations--"[w]e routinely distinguish hospitals, prisons, political parties, universities, stock exchanges, coal mines and fast-food chains" (p. 54)--but it is unclear, what a cross between a hospital and a coal mine is (for example) or how it could be represented.
Hannan and Freeman (1986) and Romanelli (1991) review several candidates for organizational features, including organizational building instructions, transactions and routines. Given a set of features, organizations can be represented using simple bit strings to encode the presence or absence of a feature. Such a representation has the advantage that it can be easily manipulated by the GA. Organizations can be easily crossed: both are split at a randomly chosen cross-over point (e.g., after the 10th bit); the first half (e.g., bits 1 to 10) of the first is concatenated with the second half of the second (e.g., bits 11 and on) and vice versa to create two new hybrid strategies. As well, solutions can be mutated by changing one or more of the bits. A significant disadvantage is that this representation often determines in advance the size and shape of the final solution, i.e., it is not sufficiently dynamic (Koza, 1992).
A second requirement is that we would prefer that the space of possible organizations be "dense", that is, modifications to the representation of one organization should usually result in another viable organization. Simply representing the formal structure of an organization might be unsatisfactory, for example, because the effects of replacing one department (marketing say) with another (manufacturing) will usually be a non-functioning organization (one with two manufacturing divisions but no marketing).
Michalewicz (1992) suggests that for the GA to perform better it is necessary to incorporate, "more problem-specific knowledge in the chromosomes' data structures" (p. 7). Bean (1994) has experimented with several techniques for mapping a dense and simple space into the desired space, e.g., mapping a sequence of N random numbers into a route for the travelling salesman's problem or a schedule for a machine shop. More complex representations, similar to the hierarchical LISP programs used by Koza, could allow for a wide variety of forms, including forms with complex internal structures, such as organizations.
Characteristics of the environment also determine in part the fitness of organizations. As with organizational forms, there are many characteristics of the environment that could be represented. Freeman and Hannan (1983), for example, examined the effects of environmental variability and patchiness. The characteristics of the environment modelled are those that are presumed to affect the performance of the organization on the task.
The environment may also include the other individuals in the population making the success of a particular individual dependent on the behaviour of the other individuals. For example, Axelrod (1987) used the GA to evolve strategies for playing prisoners' dilemma games, which we measured by their success in playing against other strategies. Such a model can exhibit behaviours that depend on the interaction between individuals, such as symbiosis or parasitism. In such a model, an individual's fitness may be defined only implicitly by its relative success in acquiring resources, as in Holland's Eco models (1992).
At this initial stage of research, I believe it is most useful to address questions for which there is already some theoretical agreement. This replication will allow us to develop some confidence in the technique before applying it to novel problems. As well, a theoretical base is necessary to suggest what features of organizations and the environment should be modelled. The experiment presented here was developed to test a proposition of Thompson's (1967) about how interdependent positions should be assigned to groups. Thompson suggested that "[i]n a situation of interdependence, concerted action comes about through coordination" (p. 55) and further notes that coordination requires decisions and communication (in varying amounts), which make coordinating costly. He therefore proposes that "[u]nder norms of rationality, organizations group positions to minimize coordination costs" (1967, Proposition 5.1, p. 57). In other words, the problem of organizing is to determine which positions should be assigned to which groups.
It is clear that we need to consider two other factors that influence organizational design and which are only implicit in Thompson's formulation. First, we need to note the benefit of coordinating. In Thompson's view, this benefit is "concerted action", which I will interpret as increased task performance. For some tasks, concerted action might not matter, and so no coordination cost is worth paying; for others, coordination might be necessary do the tasks at all. Second, there must be some restrictions on how positions are grouped. There must be a maximum group size, or else the easy answer is to simply put all positions in the same group. Similarly, there must be some restriction on the number of groups to which a position can belong, or else there could simply be one group per task. In some environments, positions can only be in one group; in others, it might actually make sense to have one group per task. Taking these factors into account, Thompson's proposition can be restated as, "under norms of rationality, organizations group positions to optimize the tradeoff between the benefits and costs of coordination, subject to limits on the size of groups and the number of groups to which a position can belong". It is this restatement that I will test in this experiment.
Table 1. Parameters of the simulation and settings in the reported experiments.
|Parameter||Setting in reported experiments|
|Number of positions||10|
|Number of groups||16|
|Number of tasks/position||10 selected (see text)|
|Interdependence between positions||0, 2, tasks overlap (see text)|
|Coordination cost||1/9 (see text)|
|Coordination benefit||-.5, 0, .3, .7, 1.2, 1.7, 2.25 (see text)|
|Mutation rate||1/40 % chance of a mutation per bit|
|Maximum and minimum normalizedfitnesses||100 and 1|
Table 2. Example of positions and tasks with 5 tasks assigned
11111 1 1 1 1 1
1 11111 1 1 1 1
11111 1 1 1 1 1
1 11 11111 1 1
1 1 1 11111 1 1
1 1 111111 1 1
1 1 11111 1 1 1
1 1 1 1 111111
11 1 1 111111
1 11 1 1 11111
Table 3. Overlap between actors' assigned tasks (# shared tasks/total # tasks) for positions in Table 2.
Relative benefit of coordinating. In order to calculate the tradeoff between the benefits and costs of coordinating, we must state them in a common metric. In this model, the fitness of each organization is defined to be the sum of the number of tasks each position performs, which in turn is proportional to the time spent on those tasks. Each position was assumed to start with an initial allocation of time. Conceptually, each position then talks to each other position in its group, which diminishes the time available to work on tasks by a constant factor per position talked to. For this experiment, this cost was fixed at 1/9 of a position's initial time so that a position that talked to every other position would have no time left to work on any tasks (i.e., a group that included all positions would get no work done).
To model the benefit of coordination, the time a position spends on a task is increased for each other position in the group that is assigned the same task. The amount of the increase is called the coordination benefit and is stated as a percentage of a position's initial allocation of time. The coordination benefit was varied from condition to condition as shown in Table 1. Negative values of the coordination benefit were included primarily as a sanity check on the algorithm.
Interdependence. For conditions with no interdependence between positions, there is no benefit to being in a group; therefore, for this condition there should be one group for each position. On the opposite extreme, when all positions perform the same tasks, the benefit for being in a group with others is high; if the benefit is high enough, one group should form that includes all positions.
Coordination benefit. When the coordination benefit is zero or negative, again, there is no benefit to being in a group, so again each position should be in its own group. On the other hand, when the coordination benefit is high, all positions should be in the same group, since the benefit will outweigh the cost. For intermediate values of the coordination benefit, groups of intermediate size should form, grouping positions with high interdependence.
Hypothesis 1: When the interdependence is 0, the number of groups will be 10.
Hypothesis 2: When the coordination benefit is negative or zero, the number of groups will be 10.
Hypothesis 3: When the coordination benefit is high and the interdependence is high, the number of groups will be 1.
Hypothesis 4: For intermediate values of interdependence and coordination benefit, groups will form that group positions with relatively high levels of interdependence.
These forces and predictions about the number of groups are summarized in Table 4.
Table 4. Summary of conditions, hypothesized forces and expected organizational forms.
|Negative or zero||Low||High|
|Inter-dependence||Favoured result||Many groups||Few groups||One group|
|Zero||Many groups||Many groups||Many groups||Many groups|
|Medium||Few groups||Many groups||Few groups||One group|
|High||One group||Many groups||Few groups||One group|
There are many variations on the GA. The one used for this paper is what Davis (1991, p. 35) describes as a "traditional GA". Fitnesses were normalized using linear normalization (Davis, 1991. p. 33), that is, organizations were sorted in decreasing order of evaluation and fitnesses assigned starting at a maximum value and decreasing linearly to a minimum value. Davis points out two benefits to the use of this technique. First, normalizing the evaluations spreads out closely spaced organizations, thus heightening the competition in a close race. Second, using linear normalization allows a "super" individual to be strongly selected, but not so strongly that it entirely dominates the population. As well, reproduction was elitist (Davis, 1991, p. 34), meaning that the two best individuals from each generation were simply copied to the next, thus preventing them from being eliminated by the vagaries of random selection. Davis notes that this strategy usually improves the performance of the GA.
Table 5. Example final organization form (5 task overlap, no coordination benefit).
Position 0, Group: 6
Position 1, Group: 0
Position 2, Group: 13
Position 3, Group: 4
Position 4, Group: 9
Position 5, Group: 15
Position 6, Group: 5
Position 7, Group: 12
Position 8, Group: 10
Position 9, Group: 2
Table 6. Average number of groups by interdependence and coordination benefit.
Table 7. Average size of largest group by interdependence and coordination benefit.
Hypothesis 2, that when the coordination benefit is negative or zero, no groups would form, seems to be supported. Again, for these conditions, the average number of groups is 10, which again is unlikely to occur by chance.
Hypothesis 3, that when the coordination benefit is high and the interdependence is high, there will be only 1 group, seems to be contradicted by the results in Table 6, which show an average around 2. Still, this number of groups is significantly fewer than expected by chance: of the 5000 randomly generated organizations, only 6 had 4 groups and none had fewer than that. Closer examination of the data shows that organizations with only 1 group are found in some but not all runs. Table 7 shows that the average size of the largest group is quite large. It appears that in the remaining runs most of the positions are gathered into one group, with a few stragglers in their own groups. It should be noted that there are only 16 organizations with 1 group, while there are 16x15x...x7 or approximately 3x1010 ways to form organizations with 10 groups. As a result, it is much easier for the simulation to find organizations that satisfy Hypotheses 1 and 2 then it is to find those for Hypothesis 3.
Additional analyses were necessary to test Hypothesis 4, that for intermediate values of interdependence and coordination benefit, groups will form that group positions with relatively high levels of interdependence. A matrix of interdependences between positions is calculated by counting the number of tasks they have in common, as in the example in Table 3. Once positions are collected in groups, a subset of the interdependence matrix can be identified that includes interdependences only between positions in the same group. To test Hypothesis 4, it is necessary to determine the similarity between these matrices. There are many ways to determine this similarity; usually a technique would be chosen based on the properties of the underlying data so statistics can be done on the measure. It is clear that the data in these matrices are not well distributed, making traditional statistics difficult, but again, traditional statistics are unnecessary since the actual distributions can be directly calculated for any measure chosen. I therefore chose to calculate the Pearson's correlation between the upper diagonals of the matrices. Table 8 shows the average correlation for the intermediate results, i.e., those with fewer than 10 but more than 2 groups; presumably any other measure would give essentially the same results.
Table 8. Average correlation between position interdependence and group assignment for intermediate conditions.
Coordination benefit Low High Interdependence .30 .70 1.20 1.70 2.25 1 .69 .74 3 .63 .72 .72 .73 5 .38 .65 .75 7 .59
The significance of this measure was determined by comparing the calculated correlation to an empirically derived distribution developed by randomly generating 5000 pairs of environment and organization for each level of interdependence and calculating the correlation between the resulting two matrices. Based on this analysis, all of these correlations are much greater than would be expected by chance; in fact, as Table 9 shows, in all but one case the significance of the minimum correlation found in the 30 runs is in the upper 5%. In the exceptional case (interdependence 5 and coordination benefit .30), the benefits of coordinating just outweigh the costs and in 5 of the 30 runs, no groups were formed, resulting in a correlation of zero which is not significant. The significance of the correlation in the remaining 25 cases is in the upper 10%. Based on these results, it appears that Hypothesis 4 is also supported.
Table 9. Significance of minimum correlation found.
Coordination benefit Low High Interdependence .30 .70 1.20 1.70 2.25 1 .96 .99 3 .97 .99 .99 1.00 5 .25 .99 .99 7 1.00The model has several possible extensions. First, positions can currently belong only to a single group; the model could be extended to allow positions to be in multiple groups simultaneously. Second, positions could be allowed to form a hierarchy, further testing Thompson's propositions.
These routines can be modelled as rules or productions indicating the appropriate action to be taken in a situation. For example, a meme could be represented as a production in a classifier system; such systems have already been successfully used with the GA (Holland, 1992). Such a representation seems particularly appropriate for use with the GA because of the claimed robustness of a rule-base to additions or deletions of individual rules. A disadvantage is that identifying routines may be problematic; as Romanelli (1991) points out, routines are "empirically elusive" (p. 87).
A second strategy would instead model the competition and cooperation between subunits within an organization to show how these interactions result in a particular organizational form--what Carroll (1984) calls the organizational level. The individuals being evolved in this approach are the components of the organization; the entire population would therefore represent a single organization. In this case, it would be desirable to develop some mechanism by which subunits could specialize for a particular role in the organization. This approach could also be used to explore the emergence of organizations by modelling interactions between independent actors; however, in this case we would need to develop criteria--perhaps related to patterns of interaction --for when an organization has emerged and which subunits are included.
Evolving individual subunits of an organization might avoid at least the first problem: most organizations (i.e., populations of subunits) will include all necessary routines, but different patterns of distribution of this know-how will result in different performance. However, to implement such a model in a GA, there must be some mechanism that distributes the payoff the organization receives among the subunits that contribute to the result. Subunits that contribute more to an organization's success can then be selected and used to breed the next generation of organizational subunits, thus changing and hopefully improving the entire organization. The distribution of these payoffs might be another way to determine which subunits are part of an emerging organization.
Each type of actor behaves in a characteristic fashion. Although Malone did not describe them in this way, we can analyze each actor's behaviour as a set of routines for primitive operations and for interacting with other actors, as shown in Table 10. Malone's analysis considered only pure organizational forms and therefore only a limited variety of organizational actors: by mixing these basic capabilities we may be able to generate a wide variety of intermediate forms, such as a cross between a processor and product manager that performs some tasks on its own and delegates others to another actor.
Table 10. Capabilities of different actor types in Malone's (1987) model organizations.
|Actor type||Capabilities and knowledge|
|Processor||Perform assigned subtasks Respond to bids in a market|
|Product managers||Decompose tasks into subtasks Know one processor for each type of subtask Communicate with processors to assign subtasks Integrate results of subtasks|
|Functional manager||Know multiple processors for one type of subtask Pick best processor for a given subtask Communicate with processors to assign subtasks|
|General manager||Decompose tasks into subtasks Know one functional manager for each type of subtask Communicate with functional manager to assign subtasks Integrate results of subtasks|
|Buyers in a decentralized market||Decompose tasks into subtasks Know multiple processors for each type of subtask Request bids for each type of subtask Evaluate bids to pick best processor for a given subtask Communicate with processors to assign subtasks Integrate results of subtasks|
|Buyers in a centralized market||Decompose tasks into subtasks Know one middleman for each type of subtask Communicate with middlemen to assign subtasks Integrate results of subtasks|
|Middlemen in a market||Know multiple processors for one type of subtask centralized Request bids for one type of subtask Evaluate bids to pick best processor for a given subtask Communicate with processors to assign subtasks|
In the GA, each actor starts with a random selection of routines and connections to other actors. As well, each actor would have a fixed set of rules for basic interactions, such as "if you want something that you know someone else has, one way to get it is to ask for it". In each generation, the behaviour of the actors is simulated and their interactions determine the performance of the organization. Actors that contribute to the success of the organization, weighted perhaps by some measure of their cost to the organization, are then selected and bred to form the next generation of actors.
Using such a model, the effect of changes in underlying parameters could be assessed. For example, Malone's models included parameters for various costs, such as performing a subtask, maintaining a unit of production capacity and sending a message and he predicted the type of organizational form that would be most efficient for different combinations of the parameters. A GA model can be validated against these predictions as well as used to identify hybrid forms that might be more appropriate for novel circumstances.
By substituting a different set of routines, entirely different types of organizations could be modelled. For example, Crowston (1991) modelled the activities performed by participants in engineering change processes and Pentland (1992) modelled the moves made by software support hotline specialists.
Two objections may be raised to this approach to the study of organizations. First, groups of positions (as in the first example) is a rather limited view of an organization. This is undeniable. Identifying appropriate tasks and organizational features is key to the utility of any kind of model. It should be noted, however, that these choices are not determined by the use of the GA, but rather depend on the organizational theories of interest. Thompson's (1967) proposition is also only about positions and groups, although a natural language presentation provides linkages to other concepts. In principle, any set of interesting features could be used, although the GA does require that they be recombinable in various ways. As the second example suggests, it may be possible to develop quite general models of organizations that can be used with a GA.
Second, even if the GA did successfully identify factors that contributed to the performance of an organizational form, it may be that the performance per se is only part of the reason for a form's success. For example, based on simulations of the evolution of competing forms, Carroll and Harrison (1992) suggest that long term success of a form may be due as much to chance as actual fitness because of the path dependent nature of the competition. Hannan and Freeman (1986) argue that institutionalization is important for the success of a form: when other powerful actors endorse a particular form's claims for resources or when it becomes unquestioned that one form is the right one to use, the difficulty of starting an organization and mobilizing resources is greatly reduced. Therefore, organizations may adopt forms without regard to their inherent performance; indeed, as Stinchcombe (1965) notes, "forms tend to incorporate and retain packages of characteristics that were fashionable or legitimate in the period when the form takes shape" (p. 53). In other words, the arrangement of positions into groups may be done for historical reasons instead of to meet the demands of the current task structure. Finally, organizational forms may differ in ways beyond those used to calculate fitness, for example, in the quality of work life they provide for their employees, how much positions holders like each other and want to be in the same groups, etc.
These caveats are certainly significant for empirical studies that attempt to explain an observed distribution of forms and such factors will certainly affect the final implementation of novel designs. For the task of suggesting new forms, however, these objections are far less damaging. Indeed, computerized implementations of organizational evolution are best seen as powerful tools for the imagination, used to help visualize novel organizational forms.
Bean, J. (1994). Genetic algorithms and random keys for sequencing and optimization. ORSA Journal on Computing, 6(2), 154-160.
Booker, L. B., Goldberg, D. E. and Holland, J. H. (1987). Classifier Systems and Genetic Algorithms (Technical Report No. 8). CSMIL, University of Michigan.
Burton, R. M. and Obel, B. (1980). A computer simulation test of the M-form hypothesis. Administrative Science Quarterly, 25, 457-66.
Carroll, G. R. (1984). Organizational ecology. Annual Review of Sociology, 10, 71-93.
Carroll, G. R. and Harrison, J. R. (1992). Chance and Rationality in Organizational Evolution (Unpublished manuscript). University of California at Berkeley.
Crowston, K. (1991). Towards a Coordination Cookbook: Recipes for Multi-Agent Action. Unpublished doctoral dissertation, MIT Sloan School of Management.
Davis, L. (Eds.). (1991). Handbook of Genetic Algorithms. New York: Van Nostrand Reinhold.
DiMaggio, P. and Powell, W. (1983). Institutional isomorphism. American Sociological Review, 48, 147-60.
Freeman, J. and Hannan, M. T. (1983). Niche width and the dynamics of organizational change. American Journal of Sociology, 88, 116-45.
Friedman, L. W. and Friedman, H. H. (1995). Analyzing simulation output using the bootstrap method. Simulation, 64(2), 95-100.
Grefenstette, J. J. (1991). Strategy acquisition with genetic algorithms. In L. Davis (Eds.), Handbook of Genetic Algorithms (pp. 186-201). New York: Van Nostrand Reinhold.
Hannan, M. T. and Freeman, J. (1986). Where do organizational forms come from? Sociological Forum, 1(1), 50-72.
Holland, J. H. (1992). Adaptation in Natural and Artificial Systems (2nd ed.). Ann Arbor, MI: University of Michigan.
Koza, J. R. (1992). Genetic Programming. Cambridge, MA: MIT Press.
Malone, T. W. (1987). Modeling coordination in organizations and markets. Management Science, 33, 1317-1332.
McKelvey, B. (1982). Organizational Systematics: Taxonomy, Evolution, Classification. Berkeley: University of California.
Michalewicz, Z. (1992). Genetic Algorithms + Data Structures = Evolution Programs. New York: Springer-Verlag.
Nelson, R. R. and Winter, S. G. (1982). An Evolutionary Theory of Economic Change. Cambridge, MA: Harvard.
Pentland, B. T. (1992). Organizing moves in software support hotlines. Administrative Science Quarterly, 37, 527-548.
Romanelli, E. (1991). The evolution of new organizational forms. Annual Review of Sociology, 17, 79-103.
Singh, J. V. and Lumsden, C. J. (1990). Theory and research in organizational ecology. Annual Review of Sociology, 17, 161-193.
Stinchcombe, A. L. (1965). Social structure and organizations. In J. G. March (Eds.), Handbook of Organizations (pp. 153-193). Chicago: Rand McNally.
Thompson, J. D. (1967). Organizations in Action: Social Science Bases of Administrative Theory. New York: McGraw-Hill.
Williamson, O. E. (1975). Markets and Hierarchies. New York: Free Press.