A problem I’ve found in modeling scholarship is how “evidence/argument words” are always highly represented in any given corpus. If you use hyperparameter optimization, which weighs topics according to the relative proportion in the corpus, words like “fact evidence argue make” tend to compose the most representative topics. Options include simply eliminating the topic from the browser, which seems to eliminate a large number of documents that would be otherwise classified, or trying to add all of the evidence words to a stop list. The aggressive pursuit of stop-words degrades the model, though this observation is more of an intuition than anything I can now document.

I thought it might be helpful to others who are interested in working with topic models to create several models of the same corpus and look at the effects created by small changes in the parameters (number of topics, lemmatization of corpus, and stop-words). The journal that I chose to use for this example is the Journal of Law and Economics, for both its ideological interest and methodological consistency. The law-and-economics movement is about as far away from literary studies as it’s possible to be while still engaging in a type of discourse analysis, I think, and I find this contrast both amusing and potentially illuminating. That the field of law-and-economics is perhaps the most well-known (even infamous) example of quantified reasoning used in support of what many view as a distinct political agenda is what led me to choose it to begin to explore the potential critical usefulness of another quantitative method of textual analysis.

I began by downloading all of the research articles published in the journal from JSTOR’s Data for Research. There were 1281 articles. I then converted the word-frequency lists to bags-of-words and created a 70-topic model using MALLET.* The browsable model is here. The first topic is the most general of academic evidence/argument words: “made, make, case, part, view, difficult. . .” I was intrigued by the high-ranking presence of articles by Milton Friedman and R. H. Coase in this topic; it would be suggestive if highly cited or otherwise important articles were most strongly associated with the corpus’s “evidence” terms, but I can’t say that this is anything other than coincidence. The next topic shows the influence of the journal’s title: “law, economics, economic, system, problem, individual.” The duplication of the adjective and noun form of “economics” can be eliminated with stemming or lemmatizing the corpus, though it is not clear if this increases the overall clarity of the model. I noticed that articles “revisiting” topics such as “social cost” and “public goods” are prominent in this topic, which is perhaps explainable by an unusually high proportion of intra-journal citations. (I want to bemoan, for the thousandth time, the loss of JSTOR’s citation data from its API.)

The next two topics are devoted to methodology. Econometric techniques dominate the content of the Journal of Law and Economics, so there’s no surprise that topics featuring those terms would be this widely distributed. Of the next three topics, one seems spuriously related to citations and the other two are also devoted to statistical methodology. It is only the eighth topic that is unambiguously associated with a recognizable subject in the journal: market efficiency. Is this apparent overemphasis on evidence/methodology a problem? And if so, what do you do about it? One approach would be to add many of the evidence-related words to a stop-list. Another would be to label all the topics and let the browser decide which are valuable. Here is a rough attempt at labeling the seventy-topic model.

The number of topics generated is the most obvious and effective parameter to adjust. Though I ended up labeling several of the topics the same way, I’m not sure that I would define those as split topics. The early evidence/methodology related topics do have slightly distinct frames of reference. The topics labeled “Pricing” also refer to different aspects of price theory, which I could have specified. The only obviously lumped-together topic was the final one, with its mixture of sex-worker and file-sharing economics. If there is evidence of both lumping and splitting, then simply adjusting the number of topics is unlikely to solve both problems.

An alternative to aggressive stop-wording is lemmatization. The Natural Language Toolkit has a lemmatizer that calls on the WordNet database. Implementation is simple in python, though slow to execute. A seventy-topic model generated with the lemmatized corpus has continuities with the non-lemmatized model. The browser shows that there are fewer evidence-related topics. Since the default stop-word list does not include the lemmatized forms “ha,” “doe,” “wa,” or “le,” it aggregates those in topics that are more strongly representative than the similar topics in the non-lemmatized model. Comparing the labeled topics with the non-lemmatized model show that there are many direct correspondences. The two insurance-related topics, for instance, have very similar lists of articles. The trend lines do not always match very well, which I believe is caused by the much higher weighting of the first “argument words” topic in the lemmatized corpus (plus also issues about the reliability of graphing these very small changes).

Labeling is inherently subjective, and my adopted labels for the lemmatized corpus were both whimsical in places and also influenced by the first labels that I had chosen. As I mentioned in my comments on Matthew Jockers’s Macroanalysis, computer scientists have developed automatic labeling techniques for topic models. While labor-intensive, doing it by hand forces you to consider each topic’s coherence and reliability in a way that might be easy to miss otherwise. The browser format that shows the articles most closely associated with each topic helps label them as well, I find. It might not be a bad idea for a topic model of journal articles to label each topic based on the title of the article most closely associated with it; this technique would only mislead on deeply divided or clustered topics, or on those which have only one article strongly associated with it (a sign of too many topics in my experience).

(UPDATE: My initial labeling of the tables below was in error because of an indexing error with the topic numbers. The correlations below make much more sense in terms of the topics’ relative weights, and I’m embarrassed that I didn’t notice the problem earlier.)

The topics were not strongly correlated with each other in either direction. In the non-lemmatized model, the only topics with a Pearson correlation above .4 were

The negative correlations below -.4 were

Ted Underwood and Andrew Goldstone’s PMLA topic-modeling post used network graphs to visualize their models and produce identifiable clusters. I suspect this particular model could be graphed in the same way, but the relatively low correlations between topics makes me a little leery of trying it. I generated a few network graphs for John Laudun’s and my folklore project, but we didn’t end up using them for the first article. They weren’t as snazzy as the Underwood and Goldstone graphs, as my gephi patience often runs very thin. (Gephi also has problems with the latest java update, as Ian Milligan pointed out to me on twitter. I intend to update this post before too long with a D3 network graph of the topic correlations.)

[UPDATE: 5/16/13. After some efforts at understanding javascript's object syntax, I've made a clickable network graph of correlations between topics in the lemmatized browser: network graph. The darker the edge, the stronger the correlation.]

The most strongly correlated topics in the lemmatized corpus were

Here is a simple network graph of the positively correlated topics above .2 (thicker lines indicate stronger correlation):

My goal is to integrate a D3.js version of these network graphs into the browsers, so that the nodes link to the topics and that the layout is adjustable. I haven’t yet learned the software well enough to do this however. The simple graph above was made using the R igraph package. [UDPATE: See here for a simple D3.js browser.]

And the negative correlations:

The fact that some topics appear at the top of both the negative and positive correlations in both of the models suggests to me that there is some artifact of the hyperparameter optimization process responsible for this in a way that I don’t quite grasp (though I am aware, sadly enough, that the explanation could be very simple). The .4 threshold I chose is arbitrary, and the correlations follow a consistent and smooth pattern in both models. The related articles section of these browsers is based on Kullback-Leibler divergence, a metric apparently more useful than Manhattan distance. It seems to me that the articles listed under each topic are much more likely to be related to one another than any metric I’ve used to compare the overall weighting of topics.

Another way of assessing the models and label-interpretations is to check where they place highly cited articles. According to google scholar, the most highly cited article** in Journal of Law and Economics is Fama and Jensen’s “Separation of Ownership and Control.” In the non-lemmatized model, it is associated with the AGENTS AND ORGANIZATIONS topic. It appears in the topic I labeled INVESTORS in the lemmatized corpus, but further reflection shows that these terms are closer than I first thought. My intuition, as I have mentioned before in this discussion of Pierre Nora’s “Between Memory and History,” is that highly cited articles are somehow more central to the corpus because they affect the subsequent distribution of terms. The next-most cited article, Oliver Williamson’s “Transaction-cost Economics: The Governance of Contractual Relations” appears, suitably enough, in the topics devoted to contracts in both browsers. And R. H. Coase’s “The Federal Communications Commission” is in the COMMUNICATIONS REGULATION topic in both browsers, a topic whose continuing theoretical interest to the journal was established by Coase’s early article.

As I mentioned in the beginning, I chose the Journal of Law and Economics for this project in interpreting topics in part because of its ideological interest. I have little sympathy for Chicago-style economics and its dire public policy recommendations, but I only expressed that in this project through some sarcastic topic-labeling. Does the classification and sorted browsing enabled by topic modeling affect how a reader perceives antagonistic material? Labeling can be an aggressive activity; would automated labeling of topics alleviate this tendency or reinforce it? I don’t know if this subject has been addressed in informational-retrieval research, but I’d like to find out.

*I am leaving out some steps here. My code that processes the MALLET output into a browser uses scripts in perl and R to link the metadata to the files and create graphs of each topic. Andrew Goldstone’s code performs much the same functions and is much more structurally sound than what I created, which is why I haven’t shared my code. For creating browsers, Allison Chaney’s topic-modeling visualization engine is what I recommend, though I was unsure how to convert MALLET’s output to the lda-c output that it expects (though doing so would doubtlessly be much simpler than writing on your own as I did).

**That is the most highly cited article anywhere that google’s bots have found, not just in the journal itself. I am aware of the assumption inherent in claiming that a highly cited article would necessarily be influential to that particular journal’s development, since disciplinary and discourse boundaries would have to be taken into account. All highly cited articles are cited in multiple disciplines, I believe, and that applies even to a journal carving out new territory in two well-established ones like law and economics.

Before I get to that, however, I want to talk about the Representations project’s status, as it was based on viewing the same corpus through a number of different topic-sizes. I had an intuition that documents that were highly cited outside of the journal, such as Pierre Nora’s “Between Memory and History,” might tend to be more reflective of the journal’s overall thematic structure than those less-cited. The fact that citation-count is (to some degree) correlated with publication date complicates this, of course, and I also began to doubt the premise. The opposite, in fact, might be as likely to be true, with articles that have an inverted correlation to the overall thematic structure possibly having more notability than “normal science.” The mathematical naivety of my approach compared to the existing work on topic-modeling and document influence, such as the Gerrish and Blei paper I linked to in the original post, also concerned me.

One important and useful feature missing from the browsers I had built was the display of related documents for each article. After spending one morning reading through early issues of Computers and the Humanities, I built a browser of it and then began working on computing similarity scores for individual articles. I used what seemed to be the simplest and most intuitive measure–the sum of absolute differences of topic assignments (this is known as Manhattan distance). Travis Brown pointed out to me on twitter that Kullback-Leibler divergence would likely give better results.* (Sure enough, in the original LDA paper, KL divergence is recommended.) The Computers and the Humanities browser currently uses the simpler distance measure, and the results are not very good. (This browser also did not filter for research articles only, and I only used the default stop-words list, which means that it is far from as useful as it could be.)

While the KL-divergence is not hard to calculate, I didn’t have time at the beginning of the end of the semester to rewrite the similarity score script to use it.** And since I wanted the next iteration of the browsers to use the presumably more accurate document-similarity scores, I’ve decided to postpone that project for a month or so. Having a javascript interface that allows you to instantly switch views between pre-generated models of varying numbers of topics also seemed like a useful idea; I haven’t seen anyone do that with different numbers of topics in each model yet (please let me know if there are existing examples of something like this).

2. Interpretation

I’m only going to write about a small section of Macroanalysis here. A full review might come in the future. I think that the rhetorical strategies of Jockers’s book (and also of Stephen Ramsay’s Reading Machines, an earlier volume in the Topics in the Digital Humanities series published by the University of Illinois Press) contrast interestingly with other scholarly monographs in literary studies and that this rhetoric is worth examining in the context of the current crisis in the humanities, and the salvific role of computational methods therein. But what I’m going to discuss here is Jockers’s take on labeling and interpreting the topics generated by LDA.

In our interpretation of the folklore-journals corpus John and I did do de-facto labeling or clustering of the topics. We were particularly interested in a cluster of topics related to the performative turn in folklore. Several of these topics did match our expectations in related terms and chronological trends. (Ben Schmidt’s cautions about graphing trends in topics chronologically are persuasive, though I’m more optimistic than he is about the use of dynamic topic modeling for secondary literature.) The documents associated with these apparently performance-related topics accorded with our expectations, and we took this as evidence that the co-occurrence and relative frequency assignments of the algorithm were working as expected. If that were all, then the results would be only another affirmation of the long-attested usefulness of LDA in classification or information-retrieval. And this goes a long way. If it works for things we know, then it works for things we don’t. And there are many texts we don’t know much about.

The real interest with using topic modeling to examine scholarship is when the results contrast with received understanding. When they mostly accord with what someone would expect to find, but there are oddities and discrepancies, we must interpret the results to determine if the fault lies in the algorithm’s classification or in the discipline’s received understanding of its history. By definition, this received understanding is based more on generalization and oral lore rather than analytic scrutiny and revision (which obviously drives much inquiry, but is almost always selective in its target), so there will always be discrepancies. Bibliometric approaches to humanities scholarship lag far behind those of the sciences, as I understand it, and I think they are of intrinsic interest independent of their contribution to disciplinary history.

Jockers describes efforts to label topics algorithmically in Macroanalysis (135, fn1). He mentions that his own work in successively revising the labels of his 19th century novels topic model is being used by David Mimno to train a classifying algorithm. He also cites “Automatic Labeling of Topic Models” and “Best Topic Word Selection for Topic Labelling” by Jey Han Lau and co-authors. Both of these papers explore automatically assigning labels to topics from either the terms themselves or from querying an external source, such as wikipedia, to correlate with the terms. My browsers just use the first four terms of a topic as the label, but I can see how a human-assigned label would make them more consistently understandable. Of course, with many models and large numbers of topics, this process becomes laborious, thus the interest in automatic assignment.

But some topics cannot be interpreted. (These are described as “uninterruptable” topics in Macroanalysis [129] in what I assume is a spell-check mistake.) Ignoring ambiguous topics is “a legitimate use of the data and should not be viewed with suspicion by those who may be wary of the ‘black box’” (130) I agree with Jockers here. In my experience modeling JSTOR data, there are always “evidence/argument” related topics that are highly represented in a hyperparametrized model, and these topics are so general as to be useless for analytic purposes. There are also “OCR error” topics and “bibliography” topics. I wouldn’t describe these latter ones as ambiguous so much as useless, but the point is that you don’t have to account for the entire model to interpret some of the topics. Topics near the bottom of a hyperparametrized model tend not to be widely represented in a corpus and thus are not of very high quality: this “dewey ek chomsky” topic from the browser I created out of five theory-oriented journals is a good example.

I was particularly intrigued by Jockers’s description of combining topic-model and stylometric classifications into a similarity matrix. I would be bewildered and intimidated by the underlying statistical difficulties of combining these two types of classifications, but the results are certainly intriguing. The immortal George Payne Rainsford James and his The False Heir was classified as the closest non-Dickens novel to A Tale of Two Cities, for example (161).

3. The JDH Issue

Scott Weingart and Elijah Meeks, as I noted above, co-edited a recent issue of JDH devoted to topic modeling in the humanities. Many of the articles are versions of widely circulated posts of the last few months, such as the aforementioned Ben Schmidt article and Andrew Goldstone’s and Ted Underwood’s piece on topic-modeling PMLA. (Before I got distracted by topic-browsers, I created some network visualizations of topics similar to those in the Underwood and Goldstone piece. I get frustrated easily with Gephi for some reason, but the network visualization packages in R don’t generally produce graphs as handsome as Gephi’s.) There is a shortened version of David Blei’s “Probabilistic Topic Models” review article, and the slides from David Mimno’s very informative presentation from November’s Topic-Modeling workshop at the University of Maryland. Megan R. Brett does a good job of explaining what’s interesting about the process to a non-specialist audience. I’ve tried this myself two or three times, and it’s much more difficult than I expected it would be. The slightly decontextualized meanings of “topic,” “theme,” “document,” and possibly even “word” that are used to describe the process cause confusion, from what I’ve observed, and it’s also quite difficult to grasp why the “bag of words” approach can produce coherent results if you’re unaccustomed to thinking about the statistical properties of language. Formalist training and methods are hard to reconcile with frequency-based analysis.

Lisa Rhody’s article describes using LDA to model ekphrastic poetry. I was impressed with Rhody’s discussion of interpretation here, as poetry presents a different level of abstraction from secondary texts and even other forms of creative writing. I had noticed in the rhetoric browser I created out of College English, jac, Rhetoric Review, Rhetoric Society Quarterly, and CCC, that the poems often published in College English consistently clustered together (and that topic would have been clustered together had I stop-worded “poems,” which I probably should have done.) Rhody’s article is the longest of the contributions, I believe, and it has a number of observations about the interpretation of topics that I want to think about more carefully.

Finally, the overview of tools available for topic modeling was very helpful. I’ve never used Paper Machines on my zotero collections, but I look forward to trying this out in the near future. A tutorial on using the R lda package might have been a useful addition, though perhaps its target audience would be too small to bother. I think I might be one of the few humanists to experiment with dynamic topic models, which I think is a useful and productive—if daunting—LDA variant. (MALLET has a built-in hierarchical LDA model, but I haven’t yet experimented with it.)

*Here is an informative storified conversation about distance measurements for topic models that Brown showed me.

**Possibly interesting detail: at no point do any of my browser-creation programs use objects or any more complicated data-structure than a hash. If you’re familiar with the types of data manipulation necessary to create one of these, that probably sounds somewhat crazy—hence my reluctance to share the code on github or similar. I know enough to know that it’s not the best way to solve the problem, but it also works, and I don’t feel the need to rewrite it for legibility and some imagined community’s approval. I’m fascinated by the ethos of code-sharing, and I might write something longer about this later.

***I disagree with the University of Illinois Press’s decision to use sigils instead of numbered notes in this book. As a reader, I prefer endnotes, though I know how hard they are to typeset, but Jockers’s book has enough of them that they should be numbered.

I used google scholar to search for most-cited articles in several journals in literary studies and allied fields. (Its default search behavior is to return the most-cited article in its database, which, while having a very broad reach, is far from comprehensive or error-free.) By far the most-cited article I found in any of the journals I looked at was Pierre Nora’s “Between Memory and History: Les Lieux de Mémoire.” A key to success in citation-gathering is multidisciplinary appeal, and Nora’s article has it. It is cited in history, literary studies, anthropology, sociology, and several other fields. (It would be interesting to consider Nora’s argument about the ever-multiplying sites of memory in era of mass quantification, but I’ll have to save that for another time.)

The next question that came to mind would be where Nora’s article would be classified in a topic model of all of the journal’s articles. Representations was first published in 1983. The entire archive in JSTOR contains 1036 documents. For many of my other topic-modeling work with journals, I have only used what JSTOR classifies as research articles. Here, because of the relatively small size of the sample (and also because I wanted to see how the algorithm would classify front matter, back matter, and the other paraphernalia), I used everything. In order to track “Between Memory and History,” I created several different models. It is always a heuristic process to match the number of topics with the size and density of a given corpus. Normally, I would have guessed that somewhere between 30-50 would have been good enough to catch most of the distinct topics while minimizing the lumping together of unrelated ones.

For this project, however, I decided to create six separate models with an incrementally increasing number of topics. The number of topics in each is 10, 30, 60, 90, 120, and 150. I have also created browsers for each model. The index page of each browser shows the first four words of each topic for that model. The topics are sorted in descending order of their proportion in the model. Clicking on one of the topics takes you to a page which shows the full list of terms associated with that topic, the articles most closely associated with that topic (also sorted in descending order—the threshold is .05), and a graph that shows the annual mean of that topic over time. Clicking on any given journal article will take you to a page showing that journal’s bibliographic information, along with a link to JSTOR. The four topics most closely associated with that article are also listed there.

In the ten-topic browser, whose presence here is intended to demonstrate my suspicion that ten topics would not be nearly enough to capture the range of discourse in Representations, Nora’s article is in the ‘French’ topic, a lumped-together race/memory topic, a generalized social/history topic, and the suggestive “time, death, narrative” topic. With a .05 threshold, 32% of the documents in the corpus appear in the ten-topic browser. [UPDATE: 3/16, this figure turned out to be based on a bug in the browser-building program.] None of these classifications are particularly surprising or revealing, given how broad the topics have to be at this level of detail; but one idea that I want to return is the ability of topic-models to identify influential documents in a given corpus. Nora’s article has clearly been very influential, but are there any detectable traces of this influence in a model of the journal in which it appeared?

Sean M. Gerrish and David Blei’s article “Language-based Approach to Measuring Scholarly Impact” uses dynamic topic models to infer which documents are (or will be) most influential in a given collection. What I have done with these Representations models is not dynamic topic modeling but the regular LDA model. I have experimented with dynamic topic models in the past, and I would like to apply the particular techniques described in their article once I can understand them better.

Here is how Nora’s article is classified in each of the topic models (sorted vertically from most to least representative):

There is a notable consistency between the topics the article is assigned to no matter how many there are to choose from. A logical question to ask is if Nora’s article is assigned to more or less topics than the average article across these six models. The percentage of all articles that are assigned to a topic with a proportional threshold >= .05 ranges from 32% with the ten-topic model to 52% in the 150-topic.

In my next post, I am going to describe the relative frequency of the average article in the different models and try to identify which ones (including Nora’s, if it turns out to be) are disproportionately represented in the topics. I will also begin interpreting these results in light of what I felt was historicism’s relative absence in the theory-journals corpus I created earlier.

[UPDATE: 3/16. I corrected a bug in the browser-building program and generated a new table above with the correct topics linked for Nora's article. The previous table had omitted a few.]

I could easily imagine someone in game studies wanting to learn how to program games in their original environment, such as 6502 assembly, for example. A good materialist impulse, such as learning how to work a printing press or bind a book, should never be discouraged. But what about scholars who have an interest in digital media, electronic editing, or text mining? The skeptical argument here points out that there are existing tools for all of these activities, and the wise and conscientious scholar will seek those out rather than wasting time reinventing an inferior product.

This argument is very persuasive, but it doesn’t survive contact with the realities of today’s text-mining and machine-learning environment. I developed a strong interest in these areas several months ago (and have posted about little else since, sadly enough), even to the point where I went to an NEH seminar on topic modeling hosted by the fine folks at the MITH. One of the informative lectures recommended that anyone serious about pursuing topic modeling projects learn the statistical programming language R and a scripting language such as python. This came as of little surprise to me as being reassured later in the evening by a dinner companion that Southerners were of course discriminated against in academia. I had begun working with topic-modeling in R packages, and a great deal of text-munging was required to assemble the topic output in a legible format. MALLET makes this easier, but there’s no existing GUI solution* for visualizing the topics (or creating browsers of them, which some feel is more useful**).

Whatever flexibility that being able to dispense with existing solutions might offer you is more than counterbalanced by the unforgiving exactitude and provincial scrupulousness of programming languages, which manifestly avoid all but the most literal interpretations and cause limitless suffering for those foolish or masochistic enough to use them. These countless frustrations inevitably lead to undue pride in overcoming them, which lead people (or at least me) to replace a more rational regret over lost time with the temporary confidence of (almost always Pyrrhic) victory.

An optimistic assessment of the future of computation is that interfaces will become sophisticated enough to eliminate the need for almost anyone other than hobbyists to program a computer. Much research in artificial intelligence (and much of the most promising results as I understand them) has been in training computers to program themselves. Functional programming languages, to my untutored eye and heavily imperative mindset, already seem to train their programmers to think in a certain way. The correct syntax is the correct solution, in other words; and how far can it be from that notable efficiency to having the computer synthesize the necessary solutions to any technical difficulty or algorithmic refinement itself? (These last comments are somewhat facetious, though the promise of autoevolution was at the heart of cybernetics and related computational enthusiasms—the recent English translation of Lem’s Summa Technologiae is an interesting source here as is Lem’s “Golem XIV.”)

I can’t help but note that several of the arguments I’ve read that advise people not to learn to code and not to spend time teaching other people how to if you happen to be unlucky enough to be in a position to do so are written by people who make it clear that they themselves know how. (I’m thinking here in particular of Brian Lennon, with whom I’ve had several discussions about these matters on twitter and also David Golumbia.) Though I don’t think this myself, I could see how someone might describe this stance as obscurantist. (It’s probably a matter of ethos and also perhaps a dislike of people who exaggerate their technical accomplishments and abilities in front of audiences who don’t know any better—if you could concede that such things could exist in the DH community.)

*Paper Machines, though I haven’t tried it out, can now import and work with DfR requests. This may include topic modeling functionality as well.

**I have to admit that casual analysis (or, exacting scrutiny) of my server logs reveals that absolutely no one finds these topic browsers worth more than a few seconds’ interest. I haven’t yet figured out if this is because they are objectively uninteresting or if users miss the links because the style sheet. (Or both.)

I’ve heard from people on twitter that it’s a widely held belief that machine-learning techniques (and, by extension, all quantitative methods) can only tell us what we already know about the texts. I admit some initial skepticism about the prevalence of this claim, but I’ve now seen more evidence of it in the wild, so to speak, and I think I understand where some of this overly categorical skepticism comes from. A test of the validity of topic modeling, for example, would be if it produces a coherent model of a well-known corpus. If it does, then it is likely that it will do the same for an unknown or unread group of texts. The models that I have built of scholarly literature from JSTOR, I can see, are thought by some of the people who’ve seen them to be well-understood corpora. If the models reflect the general topics and trends that people know from their knowledge of the field, then that’s great as far as it goes, but we’ll have to reserve judgment on the great unread.

One issue here is that I don’t think the disciplinary history of any field is well understood. Topic modeling’s disinterested aggregations have the potential to show an unrecognized formation or the persistence of a trend long-thought dormant. Clancy found some clustering of articles in rhetoric journals associated with a topic that she initially would have labeled as “expressivist” from several decades before she would expect. Part of this has to do with the eclectic nature of what’s published in College English, of course, and part has to do with the parallels between creative writing and expressivist pedagogy. But it’s the type of specific connection that someone following established histories is not likely to find.

Ben Schmidt noted that topic modeling was designed and marketed, to some degree, as a replacement for keyword search. Schmidt is more skeptical than I am of the usefulness of this higher-level of abstraction for general scholarly research. I know enough about anthropology to have my eyebrows raised by this Nicholas Wade essay on Napoleon Chagnon, for example, and I still find this browser of American Anthropologist to be a quicker way of finding articles than JSTOR’s interface. I created this browser to compare with the folklore browser* of the corpus that John Laudun and I have been working with. We wanted to see if topic models would reflect our intuition that the cultural/linguistic turn in anthropology and folklore diffused through their respective disciplines’ scholarly journals (the folklore corpus contains the journal most analogous to American Anthropologist, The Journal of American Folklore, but it also has other folklore journals as well) at the expected time (earlier in anthropology than folklore).

A very promising, to my mind, way of correlating topic models of journals is with networks of citations. I’ve done enough network graphs of scholarly citations to know that, unless you heavily prune and categorize the citations, the results are going to be hard to visualize in any meaningful way. (One of the first network graphs I created all of the citations in thirty years of JAF required zooming in to something like 1000x magnification to make out individual nodes. I’m far from an expert at creating efficient network visualizations, needless to say.) JSTOR once provided citation data through its Data for Research interface; it does not any longer as far as I know. This has been somewhat frustrating.

If we had citation data, taking two topics that both seem reflective of a general cultural/linguistic/poststructuralist influence, such as this folklore topic and this anthropological one would allow us to compare the citation networks to see if the concomitant rise in proportion was reflected in references to shared sources (Lévi-Strauss, for example, I know to be one of the most cited authors in the folklore corpus.) I would also like to explore the method described in this paper that uses a related form of posterior inference to discover the most influential documents in a corpus.**

This type of comparative exploration, while presenting an interesting technical challenge to implement (to me, that is, and I fully recognize the incommensurable gulf between using these algorithms and creating and refining them) can’t (yet) be mistaken for discovery. You can’t go from this to an a priori proof of non-discovery, however. Maybe no one is actually arguing this position, and I’m fabricating this straw argument out of supercilious tweets and decontextualized and half-remembered blog posts.

A more serious personal intellectual problem for me is that I find the dispute between Peter Norvig and Noam Chomsky to be either a case of mutual misunderstanding or one where Chomsky has by far the more persuasive case. If I’m being consistent then, I’d have to reject at least some of the methodological premises behind topic-modeling and related techniques. Perhaps “practical value” and “exploration/discovery” can share a peaceful co-existence.

*These browsers work by showing an index page with the first four words of each topic. You can then click on any one of the topics to see the full list of words associated with it, together with a list of articles sorted by how strongly they represent that topic. Clicking then on an individual article takes you to page that shows the other topics most associated with that article, also clickable, and a link to the JSTOR page of the article itself.

**The note about the model taking more than ten hours to run fills me with foreboding, however. My (doubtlessly inefficient) browser-creating scripts can take more than hour to run on a corpus of 10K documents, combined with another hour or more w/ MALLET and R–it really grinds down a person conditioned to expect instant results in today’s attention economy.

It occurred to me that a simple browser wouldn’t be that hard to build myself, so I made one for Clancy’s explorations of the rhetoric/composition journals in JSTOR and another for the theory corpus. (I did use Chaney’s CSS file.) I used my old graphs without the scatterplots layer for the theory-browser, as I didn’t want to take the time to regenerate those yet. And I’m not sure quite what’s going on with unicode/non-ASCII characters; theoretically the code I wrote should convert those properly. [UPDATE: Thanks to a pointer from Andrew Goldstone on twitter, I fixed the encoding issue. binmode, ":utf8" on all filehandles is the answer in perl at least.)

The articles shown for each topic are those that have that topic most strongly associated with them. It’s quite possible that other articles could have higher proportions but have another topic even more strongly associated with it. I should also rewrite the code so that it grabs all articles below a certain threshold of significance.

Then, there was this Metafilter post. Things I’ve discovered via Metafilter have taken away many months of work-time over the years, so the sensible thing to do would be to quit reading it. But that’s unlikely. In any case, Project Rosalind is a series of programming problems related to bioinformatics. It has the gamified features of “levels,” “badges,” “achievements,” and even, God help me, “xp.” There are a series of problems related to string processing, probability, and other topics. They have a tree-like structure, and you have to solve precursor problems before getting access to the later ones. Solving a problem involves downloading a dataset and submitting a solution within five minutes. After you’ve solved the problem, you can see the code that others have posted to solve the problem.

This feature is particularly interesting to me, as I have never really learned functional programming, so when I see solutions to problems that I have solved in perl in languages such as Haskell, Clojure, or Scala, it’s a bit easier to understand how they were put together. (Rosetta Code is another place to see programming problems solved in multiple languages.) You are allowed unlimited attempts to get the right answer, and you can see forum questions about the problem after two unsuccessful tries. (I have posted a question once—a rather idiotic question in retrospect—and I received a correspondingly withering response, whose impact I mitigated somewhat by imagining it spoken in the Comic Book Guy’s voice.)

I have, at this point, solved twenty-two of the ninety-three problems. The early ones are trivial, but I’m finding the difficulty to be scaling up quite a bit. I’ve used some algorithms I had never worked with before, such as tree-suffix and shortest-superstring. I’ve also used arbitrarily nested loops in perl (with Algorithm::Loops) and contemplated the theoretical limits of what a regular expression can match more than I’ve had to before. It’s also quite interesting to see what the total numbers of problems solved reveal about people’s background knowledge. Two of the problems involving Mendelian inheritance and probability have been solved proportionally many fewer times than (more difficult) string-processing programs. (I don’t mean to be a hypocrite in saying this, as I got tired of the Punnett-squares required in the second one of those and haven’t solved it myself.)

Some of the gamified features of the site I regard as silly (levels, xp, badges, achievements), but I admit that I can’t help but be motivated by the statistical information about how many people have solved which problems. It triggers my instinctual competitiveness, somehow. They even seem to encourage people to post their country of origin to introduce nationalism into the competitive mix here. As a learning tool, I’m not sure how effective it is. It’s quite possible to solve many of the problems while retaining only the barest minimum about the underlying molecular biology, and problems which require a bit more conceptual understanding than that (see the Mendelian inheritance ones above) are comparatively ignored.

The programmatic checking of solutions is also somewhat finicky. An end-of-line character at the end of the file will cause an otherwise correct solution to fail for at least some of the problems, for example. But all in all, I’m very impressed with this site and think it has a lot of potential in teaching people (humanists, for example), how to program. It would be nice to be able to reuse the code with different problem sets, if they ever decide to release the source in the future.

UPDATE:

I corrected a few mistakes (I gave myself an extra problem, for instance), and I also wanted to mention an important precursor: Project Euler. This site has mathematics problems, and it also seems a bit more streamlined. I haven’t actually used it yet, though.

I downloaded all of the articles (word-frequency data for each article, that is) in New Literary History, Critical Inquiry, boundary 2, Diacritics, Cultural Critique, and Social Text. I then ran a model fitted to one-hundred topics. I had to adjust the stop-word list to account for common words and, unsuccessfully, for words in other languages. What I should have done was use the supplied stop-word lists in those languages as well. At least this way there is a chance that interesting words in those languages will cluster together.

The topics themselves looked good, I thought. One hundred was about the right number, as I didn’t see much evidence of merging or splitting. I should say rather that I saw an acceptable level, or the usual level. This topic, for example, shows what I mean: “aboriginal rap[?] women australian climate weather movement work warming time australia housework change social power oroonoko[?] make wages years.” I also didn’t lemmatize this corpus, although I know how to. Lemmatizing takes a lot of time the way I’m doing it (using the WordNet plugin of the python Natural Language Toolkit). And I frankly haven’t been that impressed with the specificity of the lemmatized models that I have run.

Visualizing changes in topics over time is quite difficult. Each year will have thousands of observations per topic and taking the mean of each topic per year doesn’t always produce very readable results. Benjamin Schmidt suggested trying the geom_smooth function of ggplot2, which I never had much luck with. The main reason, I found, that I couldn’t get it to work very well is that I was trying to create a composite graph of every topic using facet_wrap. Each topic graphed by itself with geom_smooth produced better results.

Here, for example, is the graph for this coherent topic—”gay sexual queer sex lesbian aids sexuality homosexual men homosexuality identity heterosexual male gender desire social lesbians drag butler”:

The chronology you see above does approximately track the rise of queer theory, though the smoothing algorithm is full of mystery and error. A scatter-plot of the same graph would be far noisier and also not reveal much in the way of change over time. This topic should also correlate somewhat roughly to postcolonial theory–”indian india hindu colonial postcolonial subaltern british indians nationalist gandhi english bengali religious caste nationalism sanskrit maori bengal west”:

I’m suspicious of this linear increase, needless to say. The underlying data is messier. Would Marxist theory show any decline around the predictable historical period? (Terms: “social class theory ideology political production ideological historical marxist marx bourgeois capitalist society capitalism marxism economic labor relations capital”)

That is roughly what I was expecting. But compare “soviet party revolutionary socialist revolution socialism communist political national left union struggle europe russian fascism war central movement european”:

I have hopes for the exploratory potential of topic-modeling disciplinary change this way. Another interesting topic that shows a linear-seeming increase (“muslim islamic islam religious arab muslims secular arabic algerian orientalism rushdie religion iranian iran western turkish ibn secularism algeria”):

To show what the data looks like with different visualizations, I’m going to cycle through several types of graphs of the above topic. The first is a line graph:

Next is a scatter-plot:

Now a scatter-plot with the scale_y_log10 function applied:

And a yearly mean:

Finally, a five-year mean:

All of the graphs reveal a general upward trend, I think, though not as much as the smoothing function does. I would be delighted in hearing any ideas anyone has about better ways to graph these. I’ve not found any improvements in grouping by document rather than year.

There’s more I plan to do with this data set, including coming up with better ways to visualize it (more precision, efficient ways of seeing many at once, etc.) I am including the full list of topics after the fold for reference. Some reveal OCR errors; others are publishing artifacts that my first rounds of stopping didn’t yet remove.

Update (2/14/12): I created a browser of this model that shows the articles most closely associated with each topic.

0 0.0321 american left radical political movement social economics black time years war power orwell decade began america back students books
1 0.02376 chinese china western hong cultural kong wang boundary modern zhang west mao lu intellectual shanghai intellectuals japanese east liu
2 0.09169 french france paris pierre jean barthes bataille flaubert work proust sartre jacques louis revolution text marcel madame histoire georges
3 0.03662 movement movements left political radical american revolution cultural world aronowitz issue civil change issues society sexual freedom social history
4 0.18127 language speech words word linguistic translation english voice meaning discourse speaking speaker act speak sentence utterance languages spoken verbal
5 0.02654 woolf virginia jane beckett lentricchia gilbert austen lawrence moore richards eve forster adam room samson edna shaw bloomsbury stevens
6 0.03491 asian american united ethnic pacific states immigrant immigrants immigration racial transnational border korean diaspora hawaiian mexican chicano identity diasporic
7 0.68931 power suggests figure authority text version rhetoric appears makes irony offers terms force calls rhetorical cited earlier act ironic
8 0.03294 latin spanish cuban don cuba puerto juan borges spain america mexico mexican brazilian rican american jose brazil maria garcia
9 0.0539 medieval middle oral latin literary ages ancient tradition written auerbach century classical texts renaissance modern rhetoric augustine early philology
10 0.12145 south national state political government people rights nation community local international official africa human african policy population land resistance
11 0.04518 medical health body medicine disease aids drug illness mental patients treatment patient clinical healing hysteria madness addiction bodies coffee
12 0.05613 german das den germany als berlin ist kafka benjamin ein karl eine trans ich mit dem ernst friedrich sich
13 0.15228 social class theory ideology political production ideological historical marxist marx bourgeois capitalist society capitalism marxism economic labor relations capital
14 0.18724 philosophy theory philosophical knowledge truth thought science scientific world wittgenstein epistemological human idea philosophers view language theoretical reason empirical
15 0.21737 university trans john david duke cambridge boundary chicago michael robert harvard oxford richard paul modern james minnesota princeton peter
16 0.0917 aesthetic art benjamin adorno sublime aesthetics work kant experience critique object beautiful form modern beauty concept judgment modernity trans
17 0.12624 narrative story narrator narratives events stories narration time plot tale event voice literary telling structure discourse action account history
18 0.20746 history historical past time present historians historian future modern events period histories century human historiography temporal study change historicism
19 0.0124 aboriginal rap women australian climate weather movement work warming time australia housework change social power oroonoko make wages years
20 0.02402 jewish palestinian arab israel israeli jews palestinians palestine zionist jew arabs state zionism middle west land political east hebrew
21 0.56336 form work general structure individual forms elements analysis principle formal specific works terms single set style function features type
22 0.16719 public york media television news national american times show president united march recent audience private april states people campaign
23 0.11547 identity postmodern cultural politics postmodernism difference discourse social culture power dominant practices identities world resistance discourses history struggle language
24 0.41333 critical studies work cultural theory critique political contemporary essay recent intellectual theoretical ways historical questions analysis question practice discussion
25 0.06531 japanese money economic market economy exchange japan corporate commodity capital financial business capitalism consumption consumer commodities production economics wealth
26 0.09897 sexual women male female woman men body gender sex feminine sexuality desire masculine power man difference masculinity pleasure bodies
27 0.06501 women feminist feminism feminists gender female woman male men sexual patriarchal sex work politics political radical mary history movement
28 0.66109 argument make position evidence response good view critics find simply claim general values arguments claims problem easily issue difficult
29 0.07487 poem romantic poetry poet wordsworth poetic milton poems coleridge william mind nature yeats paradise poets blake thy shelley bloom
30 0.01992 indian india hindu colonial postcolonial subaltern british indians nationalist gandhi english bengali religious caste nationalism sanskrit maori bengal west
31 0.08238 fig photography photograph photographs figure photographic museum portrait picture objects visual pictures image images camera object medium display portraits
32 0.1694 moral human ethical freedom life ethics individual action good morality values person social nature actions reason responsibility man judgment
33 0.20092 literary literature criticism critical history critics theory critic works study language english art tradition texts modern aesthetic writers essays
34 0.14431 writing book writer life writers write read reading written books work autobiography literary autobiographical literature personal wrote style reader
35 0.01414 chomsky dewey ek war read goodman american politics marcuse movement work political state left social radical approach public life
36 0.05149 literary history cited univ text cal notes ity dis ence human ness pro sion tional term form inter comparison
37 0.06194 music musical sound jazz song dance performance sounds voice musicians songs listening play hear art playing singing radio recording
38 0.1099 american america united states national americans war john world william james henry culture north cultural cold history canadian melville
39 0.03316 gay sexual queer sex lesbian aids sexuality homosexual men homosexuality identity heterosexual male gender desire social lesbians drag butler
40 0.17065 death body violence human dead life animal living bodies man fear pain kill blood horror murder animals scene violent
41 0.09262 city space urban spatial building place spaces cities architecture site home center house landscape architectural public places built map
42 0.02614 williams james brown tom fuller eliot maggie american john bishop read margaret people smithson act charlotte book bowl robert
43 0.05497 foucault life power deleuze everyday michel modern sovereignty state political agamben trans sovereign body guattari human discipline politics disciplinary
44 0.18385 love father family marriage woman life man young mother wife story home house husband daughter desire lady women scene
45 0.69503 time order question place point longer end moment truth means present fact back precisely word speak give remains beginning
46 0.30084 water earth land sea sun place green landscape tree river sky trees snow space light stone red white high
47 0.10127 law legal rights state court justice property laws case act authority decision system rule states sovereign contract rules sovereignty
48 0.49656 desire moment return loss form death lost presence absence condition remains sense figure passage identity past end crisis sign
49 0.16624 system time systems theory information communication cognitive affect body processes affective human process temporal level space perception brain distinction
50 0.42525 world experience life human reality consciousness nature process mind imagination vision sense language individual personal meaning man form act
51 0.40925 book published years work text letter written early author title letters books publication read readers english number found wrote
52 0.06248 technology information media computer technological digital technologies electronic machine communication human control world technical data virtual machines web internet
53 0.06998 police crime trial violence criminal prison case murder legal evidence crimes political violent victim victims serial secret justice eichmann
54 0.3546 subject discourse order relation space form discursive object difference practice subjectivity logic place symbolic subjects production position mode boundary
55 0.02432 emerson ellison burke twain hawthorne trilling read invisible ralph huckleberry writers work finn jim black social literature women john
56 0.03159 renaissance pastoral king court courtly queen literary english greenblatt prince sidney elizabethan good marie royal sir henry knight text
57 0.09066 freud psychoanalysis psychoanalytic desire unconscious lacan theory object subject freudian ego psychic symbolic sexual pleasure psychological dream psychology fantasy
58 0.57855 great modern century man life made time age men history years early thought intellectual long world found nineteenth period
59 0.14516 labor economic workers work class social working economy state welfare system industrial market percent capital control poor government union
60 0.1693 text reading literary interpretation meaning texts reader textual work author interpretive readers theory history intention read understanding act interpretations
61 0.02364 muslim islamic islam religious arab muslims secular arabic algerian orientalism rushdie religion iranian iran western turkish ibn secularism algeria
62 0.07184 play theater drama audience dramatic stage performance shakespeare plays theatrical action history hamlet tragedy characters comedy comic character actor
63 0.04363 memory trauma past holocaust memories traumatic event jews nazi history experience truth jewish testimony auschwitz german victims witness war
64 0.18222 image representation images visual space body gaze object vision presence mirror representations represented eye visible point perception represent picture
65 0.06115 film films cinema camera cinematic hollywood movie screen shot documentary frame scene movies early spectator visual images time sequence
66 0.13551 inquiry critical winter autumn abbreviated professor response spring account summer trans claim chicago fact theory point made convention essay
67 0.07161 black white african racial race blacks slave negro racism slavery bois color racist whiteness whites people social class blackness
68 0.11013 metaphor language meaning sign linguistic semiotic semantic signs system theory metaphors word metaphorical structure discourse level words literal semiotics
69 0.04175 ou comme nous mais tout sans akhmatova avec cette sont baudelaire ii fait ses ces aux lydia meme leur
70 0.04025 natural science mathematical scientific mathematics machine physics quantum hobbes human descartes nature mechanical machines body universe set history einstein
71 0.10857 god religious christian religion divine church christianity secular faith theological christ theology jesus sacred spiritual holy biblical tradition jewish
72 0.09419 war military vietnam world united states american nuclear enemy terror cold power peace soldiers torture wars army violence bush
73 0.09257 art painting work artist artistic artists works arts paintings visual aesthetic modernist modernism painter modern museum abstract fried style
74 0.06986 science human scientific nature natural scientists genetic species biological environmental research evolutionary sciences knowledge evolution biology humans life environment
75 0.0255 gramsci italian italy che prison antonio fascist ii notebooks croce giovanni vita canto carlo dante verdi michelangelo rome trans
76 0.21001 people time years good talk lot back things talking women wanted feel work make told interview thing men put
77 0.19357 political politics state public social liberal power democratic society democracy civil freedom sphere people discourse rights private radical intellectuals
78 0.10585 derrida text man reading deconstruction writing language texts jacques deconstructive miller play figure read difference question rhetoric paul essay
79 0.05332 greek tragedy classical tragic ancient oedipus aristotle plato epic socrates homer roman greeks riddle greece city antigone gods athens
80 0.07717 english british irish england london john eighteenth ireland britain victorian william century early thomas sir george late history charles
81 0.09257 colonial european western native west culture world indian cultural africa african peoples imperial discourse europe indigenous anthropology people colonialism
82 0.0366 genre bakhtin genres russian generic dialogue carnival dialogic literary mikhail literature rabelais poetics dostoevsky text speech work theory pushkin
83 0.11295 myth ritual sacred myths king mythic symbolic story magic man traditional hero tale great stories power tales gods ancient
84 0.01573 shame movement black social trip individual larkin youth political white culture lsd term anger person usage heavy cultural man
85 0.10016 poetry poem poems poet poetic poets pound language poetics verse lyric line lines olson words word prose robert form
86 0.72608 kind sense things make work part thing point find fact made world making makes call ways great called sort
87 0.76598 point terms fact question notion problem sense concept part discussion view relation relationship important nature context makes idea simply
88 0.08277 soviet party revolutionary socialist revolution socialism communist political national left union struggle europe russian fascism war central movement european
89 0.10262 global world national cultural postcolonial nation international modernity globalization nationalism states united local transnational economic culture political western capital
90 0.11165 fiction novels literary characters fictional character story reader author joyce narrator readers literature romance james genre narrative works fictions
91 0.03936 game play games players sports playing player baseball sport chess world team rules ball leisure cricket played life living
92 0.05541 diacritics trans derrida ofthe time community relation levinas ethical ethics event gift possibility jacques responsibility work logic blanchot future
93 0.0573 children child family mother parents abuse birth mothers adult adoption childhood baby families maternal father kinship reproductive motherhood home
94 0.12026 university students education academic faculty student research teaching graduate humanities higher knowledge professional school universities studies educational college english
95 0.3797 social society cultural forms individual group ways relations role practices groups community people individuals power public important process life
96 0.24791 young man room street small years day side big home food boy front days car back girl began middle
97 0.09584 heidegger nietzsche thought hegel philosophy time world thinking truth trans spirit philosophical metaphysical essence sense phenomenology existence metaphysics man
98 0.47688 back eyes time face night left hand head day man hands dark light dream words house inside world black
99 0.16072 culture cultural popular mass rock high class contemporary youth production everyday industry dominant cultures american style traditional media consumption

It wasn’t. It didn’t take me too long to figure out that I needed to change the data from discrete to continuous in order to see anything like a sparkline, but it was also apparent from the other data sets I was working with that taking the mean at intervals was the only way to make a reasonably clean graph. I ended up using the aggregate function to create the n-year averages, though I read some intriguing descriptions of the power of data.tables in R. (I refuse to ask for help on stackoverflow, even though it would have saved many hours worth of work. Character flaw.)

I now need to learn how to use the reshape package, with its wonderfully named ‘melt’ and ‘cast’ features, to rewrite the code I’m using to change rows to columns. A simple for-loop iteration over a data-frame in R can take hours, I’ve learned; and I expect that this other solution would finish the job in seconds.

Anyway, here’s the revised graph of ELH with annual means of topic-proportions:

The full list of topics can be found in my previous post.

ELH I thought would be an interesting journal to try, as it publishes articles in each issue on quite a range of literary periods, often ranging from medieval to twentieth-century material. I assumed that LDA would be likely to identify each of these periods as a topic. To test this, I downloaded the entire set of articles from JSTOR and created a fifty-topic model. From there, I wanted to chart the proportion of each topic in each document. I was able to import the data in R and use ggplot2 to create the following graph:

As you can see, many of these topics are identifiable from even two-word samples. Others show a need of lemmatizing (a slow process using the python NLTK, though effective), or of further splitting. Perhaps fifty topics is not quite enough.

The code has to transform row data to column-form in order to be efficiently sorted. It then used the ggplot2 facet_wrap feature to create the graph. I’d be happy to share it, if anyone’s interested, though it uses a for-loop, which I understand to be bad R. You also have to pre-process the JSTOR files to associate dates with the files themselves. I have a perl script for this.

For reference, here is the complete list of topics generated by MALLET:
0 love marriage lady lover woman desire young lovers passion wife sexual beauty friendship husband heart story loves relationship world
1 women female sexual male woman gender men sexuality desire feminine masculine mary sex mother lady patriarchal early domestic feminist
2 place house back scene light great description eyes passage water space sea makes night earth city landscape man day
3 body human bodies medical scientific science physical natural bodily disease nature medicine mental health physiological early yellow james john
4 renaissance english modern book bacon early thomas latin humanist elizabethan classical utopia richard sixteenth cambridge england tudor erasmus knowledge
5 world life experience human sense reality mind personal feeling consciousness imagination real vision man modern individual emotional felt feel
6 medieval english middle arthur piers green poem gawain knight play poet lancelot bat st late courtly sir hym plowman
7 american war whitman poe america political conrad public literature jim world secret adams york marlow walt united leaves german
8 moral man virtue social human character fielding nature good natural society characters sentimental sympathy hero morality tom irony action
9 yeats keats tennyson marvell poem garden art flowers herrick victorian andrew poet nymph stanza beauty swinburne idylls green myth
10 wordsworth coleridge romantic byron blake poem poetry poetic william lyrical poet romanticism prelude lines nature imagination mind book wordsworthian
11 shakespeare play hamlet scene king dramatic tragedy richard othello plays macbeth action audience act shakespearean speech tragic drama measure
12 shelley political burke revolution french mary sublime caleb revolutionary radical rousseau godwin romantic historical wollstonecraft falkland reform prometheus frankenstein
13 chaucer tale troilus medieval tales canterbury prologue wife criseyde man book fortune courtly nat pardoner story knight ye lydgate
14 social literary cultural history historical culture political text form modern discourse literature work forms individual critique critical texts reading
15 johnson pope swift dryden satire addison gulliver satiric augustan wit boswell samuel essay restoration eighteenth lines spectator satirist poem
16 narrative story narrator fiction reader history characters plot tale book romance events novels readers character stories truth fictional text
17 irish scott historical national ireland gothic scottish history english british nation waverley scotland past castle novels ancient family antiquarian
18 church religious catholic protestant religion puritan john england english reformation bishop body roman anglican ecclesiastical christian st real argument
19 language words word speech meaning text reading reader writing rhetorical linguistic style voice verbal rhetoric read speak discourse sense
20 law legal family clarissa father pamela marriage property richardson lovelace child incest daughter letter rape contract lady criminal miss
21 spenser faerie book queene allegory pastoral canto knight allegorical guyon poem colin arthur red britomart venus poet books nature
22 death life time past dead nature man memory present loss child living world natural end soul mother back voice
23 sonnet sonnets line english music lines verse song musical form lyric sound rhyme sequence italian songs stanza lyrics opera
24 woolf public lamb virginia forster society burney social room miss goldsmith evelina elia bloomsbury lily young sheridan peter house
25 english british colonial european england national crusoe imperial cultural island empire indian early foreign spanish east trade west india
26 nature human man mind natural reason world things theory thought truth ideas knowledge philosophy idea philosophical object form imagination
27 jane dickens victorian lucy austen novels david charlotte bleak miss pip sir trollope catherine wuthering emma bronte fanny lady
28 literary english century literature criticism critical history works poetry critics great writers essay art modern work eighteenth age influence
29 make good made man great life men end give put find true time left found long mind thought things
30 black white american slave hawthorne racial slavery race african melville slaves baldwin negro scarlet identity ahab hester southern sentimental
31 social economic class money society labor economy public market trade commercial poor exchange domestic wealth system private property city
32 part point view made general important time fact work kind present earlier sense passage effect form make similar found
33 joyce stephen hardy wilde bloom ulysses james tess molly artist portrait young finnegans wake ford jude chapter dorian father
34 poem poetry poet poems poetic speaker lines poets line verse stanza stevens lyric work reader thy song williams elegy
35 letter book published letters edition writing written years john printed text literary early time william wrote books work author
36 eliot george pater jewish james victorian daniel jews henry gwendolen deronda marius life jew dorothea social adam maggie middlemarch
37 dracula sterne animal tristram beckett stein shandy animals stoker yorick henry horses lucy smart murphy journey dogs mechanical uncle
38 marlowe faustus ovid epic classical virgil tamburlaine dido chapman aeneas ovidian roman hercules gods georgic hero myth aeneid georgics
39 makes power place question order suggests simply means act response relationship fact terms sense role identity claim precisely critics
40 sidney elizabethan sir pastoral lady queen elizabeth essex beowulf stella court philip arcadia ralegh earl countess sonnet poet lord
41 god christian christ spiritual religious divine man grace st holy biblical faith john bible soul sin church word doctrine
42 desire figure body object subject image text violence power representation narrative scene form relation moment pleasure trans fantasy gaze
43 political king james english royal power history john state england henry charles government politics civil court war lord public
44 image world time form vision images order structure pattern symbolic meaning movement symbol figure imagery final process physical metaphor
45 milton paradise adam god satan lost eve samson book fall poem heaven epic son evil hell divine fallen sin
46 art pound aesthetic browning painting work thoreau visual ruskin blake artist cantos ezra plate aesthetics arts museum canto paintings
47 donne hath thy doth thou john doe good elizabethan henry haue owne made sir man renaissance bee thomas world
48 play plays stage jonson drama theater audience theatrical dramatic performance comedy masque sir restoration comic theatre scene actors ben
49 translation french latin il hebrew und se di ne cf ut english dans die version sed renaissance par quod