Reformatted from the original site.
No ideas but in things.
William Carlos Williams
As discussed in several of the contributed articles in this book, the Turing test was devised by Alan Turing as a way of certifying machine intelligence. Turing described a situation in which a human judge communicates with both a computer and a human using a computer terminal. The judge's task is to determine which is which. The judge cannot see the computer or the human and must make his or her determination by interviewing both. The computer attempts to trick the judge into selecting it as the human.
The essence of the Turing Test is that the computer attempts to act like a human within the context of an interview over terminal lines. A narrower concept of a Turing test is for a computer to successfully imitate a human within a particular domain of human intelligence. We might call these domain-specific Turing tests. One such domain-specific Turing test, based on a computer's ability to write poetry, is presented here.
The Kurzweil Cybernetic Poet is a computer program (written by the author) and provided with an input file of poems written by a human author or authors. The program analyzes these poems and creates a word-sequence model based on the poems it has just read. It then writes original stanzas of poetry using the model it has created. Some of the following stanzas of poetry were written by the Kurzweil Cybernetic Poet. Some were written by human authors (in fact the same human authors that were read and analyzed by the Kurzweil Cybernetic Poet). See if you can tell which are which. On a piece of paper, write down the numbers 1 through 28. Then put a C by that number if you believe that the corresponding stanza was written by the computer. Put an H if you believe the stanza was written by a human poet. The answers are in a footnote. Following the answers are the results obtained when the test was taken by 16 human (both adult and child) judges.
The Test
The above 28-question poetic Turing test was administered to 16 human judges with varying degrees of computer and poetry experience and knowledge. The 13 adult judges scored an average 59 percent correct in identifying the computer poem stanzas, 68 percent correct in identifying the human poem stanzas, and 63 percent correct overall. The three child judges scored an average of 52 percent correct in identifying the computer poem stanzas, 42 percent correct in identifying the human poem stanzas, and 48 percent correct overall.
The charts below show the actual scores obtained by the 16 human judges as broken down by adult/child, computer experience, and poetry experience. As can be seen from the charts, there were no trends based on level of computer experience or poetry experience clearly discernible from this limited sample. The adults did score somewhat better than the children. The children scored essentially at chance level (approximately 50 percent) and the adults achieving slightly better than chance.
The second chart shows the number of correct and incorrect answers for each of the 28 poems or stanzas. While the adult judges scored somewhat better than chance (63 percent), their answers were far from perfect. The computer poet was able to trick the human judges much of the time. Some of the computer poems (numbers 15 and 28, for example) were particularly successful in tricking the judges.
We can conclude that this domain-specific Turing test has achieved some level of success in tricking human judges in its poetry-writing ability. A more difficult problem than writing stanzas of poetry is writing complete poems that make thematic, syntactic, and poetic sense across multiple stanzas. A future version of the Kurzweil Cybernetic Poet is contemplated that attempts this more difficult task. To be successful, the models created by the Cybernetic Poet will require a richer understanding of the syntactic and poetic function of each word.
Even the originally proposed Turing test involving terminal interviews is notably imprecise in determining when the computer has been successful in imitating a human. How many judges need to be fooled? At what score do we consider the human judges to have been fooled? How sophisticated do the judges need to be? How sophisticated (or unsophisticated) does the human foil need to be? How much time do the judges have to make their determination? These are but a few of the many questions surrounding the Turing test. It is clear that the era of computers passing the Turing test will not happen suddenly. Once computers start to arguably pass the Turing test, the validity of the tests and the testing procedures will undoubtedly be debated. The same can be said for the narrower domain-specific Turing tests.
We have not yet reached the point at which computers can even arguably pass the originally proposed terminal-interview type of Turing test. This test requires a computer to master too many high-level cognitive skills in a single system for the computer of today to succeed. As Dan Dennett points out in his article, the unadulterated Turing test is far more difficult for a computer to pass than any more restricted version. We have, however, reached the point where computers can successfully imitate human performance within narrowly focused areas of human expertise. Expert systems, for example, are able to replicate the decision-making ability of human professionals within an expanding set of human disciplines. In at least one controlled trial, human chess experts were unable to distinguish the chess-playing style of more sophisticated computer chess players from that of humans. Indeed, computer chess programs are now able to defeat almost all human players, with the exception of a small and diminishing number of senior chess masters. Music composed by computer is becoming increasingly successful in passing the Turing test of believability. The era of computer success in a wide range of domain-specific Turing tests is arriving.
Level of computer experience | ||||
Level of poetry experience | Little | Moderate | Professional | Average |
Little | 56 | 44, 69, 75 | 63, 75 | 64 |
Moderate | 50, 56, 63 | 56, 63 | 75 | 61 |
A lot | 25 | 25 | ||
Average | 56 | 61 | 59 | 59 |
Scores | 38, 50, 69 |
Average | 52 |
Level of computer experience | ||||
Level of poetry experience | Little | Moderate | Professional | Average |
Little | 83 | 58, 58, 100 | 50, 67 | 69 |
Moderate | 60, 67, 83 | 58, 83 | 92 | 74 |
A lot | 25 | 25 | ||
Average | 73 | 72 | 59 | 68 |
Scores | 33, 42, 50 |
Average | 42 |
Level of computer experience | ||||
Level of poetry experience | Little | Moderate | Professional | Average |
Little | 68 | 50, 64, 86 | 57, 71 | 66 |
Moderate | 55, 61, 71 | 61, 68 | 82 | 66 |
A lot | 25 | 25 | ||
Average | 64 | 66 | 59 | 63 |
Scores | 39, 43, 61 |
Average | 48 |
Poem stanza | No. right | No. wrong | Computer or human poem stanza |
1 | 9 | 7 | computer |
3 | 11 | 5 | computer |
4 | 8 | 8 | computer |
6 | 9 | 7 | computer |
8 | 11 | 5 | computer |
11 | 11 | 5 | computer |
13 | 8 | 8 | computer |
14 | 10 | 6 | computer |
15 | 6 | 10 | computer |
16 | 10 | 6 | computer |
19 | 9 | 7 | computer |
20 | 12 | 4 | computer |
23 | 9 | 7 | computer |
25 | 8 | 8 | computer |
26 | 11 | 5 | computer |
28 | 6 | 10 | computer |
Average | 58% | 42% | |
2 | 9 | 7 | human |
5 | 9 | 7 | human |
7 | 9 | 7 | human |
9 | 13 | 3 | human |
10 | 8 | 8 | human |
12 | 10 | 6 | human |
17 | 9 | 7 | human |
18 | 14 | 2 | human |
21 | 11 | 5 | human |
22 | 11 | 5 | human |
24 | 11 | 5 | human |
27 | 8 | 8 | human |
Average | 64% | 36% | |
Overall average | 61% | 39% |
Four human poets were used: three famous poets (Percy Bysshe Shelley, T.S. Eliot, and William Carlos Williams) and one obscure poet (Raymond Kurzweil). In the case of the famous human poets, stanzas were selected from their most famous work. In all cases, the stanzas selected did not require adjacent stanzas to make thematic or syntactic sense. The computer stanzas were written by the Kurzweil Cybernetic Poet after it had read poems by these same human authors.
Poem stanza 1 written by the Kurzweil Cybernetic Poet
after reading poems by William Carlos Williams
Poem stanza 2 written by William Carlos Williams
Poem stanza 3 written by the Kurzweil Cybernetic Poet
after reading poems by Percy Bysshe Shelley
Poem stanza 4 written by the Kurzweil Cybernetic Poet
after reading poems by T.S. Eliot and Percy Bysshe Shelley
Poem stanza 5 written by Raymond Kurzweil
Poem stanza 6 written by the Kurzweil Cybernetic Poet
after reading poems by T.S. Eliot, Raymond Kurzweil, Percy Bysshe Shelley,
and William Carlos Williams
Poem stanza 7 written by T.S. Eliot
Poem stanza 8 written by the Kurzweil Cybernetic Poet
after reading poems by Raymond Kurzweil and T.S. Eliot
Poem stanza 9 written by William Carlos Williams
Poem stanza 10 written by Raymond Kurzweil
Poem stanza 11 written by the Kurzweil Cybernetic Poet
after reading poems by Raymond Kurzweil and T.S. Eliot
Poem stanza 12 written by William Carlos Williams
Poem stanza 13 written by the Kurzweil Cybernetic Poet
after reading poems by Percy Bysshe Shelley
Poem stanza 14 written by the Kurzweil Cybernetic Poet
after reading poems by T.S. Eliot and Percy Bysshe Shelley
Poem stanza 15 written by the Kurzweil Cybernetic Poet
after reading poems by Raymond Kurzweil and William Carlos Williams
Poem stanza 16 written by the Kurzweil Cybernetic Poet
after reading poems by T.S. Eliot, Raymond Kurzweil, Percy Bysshe Shelley,
and William Carlos Williams
Poem stanza 17 written by William Carlos Williams
Poem stanza 18 written by Raymond Kurzweil
Poem stanza 19 written by the Kurzweil Cybernetic Poet
after reading poems by T.S. Eliot, Raymond Kurzweil, Percy Bysshe Shelley,
and William Carlos Williams
Poem stanza 20 written by the Kurzweil Cybernetic Poet
after reading poems by Raymond Kurzweil and T.S. Eliot
Poem stanza 21 written by T.S. Eliot
Poem stanza 22 written by T.S. Eliot
Poem stanza 23 written by the Kurzweil Cybernetic Poet
after reading poems by Raymond Kurzweil and William Carlos Williams
Poem stanza 24 written by William Carlos Williams
Poem stanza 25 written by the Kurzweil Cybernetic Poet
after reading poems by T.S. Eliot, Raymond Kurzweil, Percy Bysshe Shelley,
and William Carlos Williams
Poem stanza 26 written by the Kurzweil Cybernetic Poet
after reading poems by Raymond Kurzweil and William Carlos Williams
Poem stanza 27 written by Percy Bysshe Shelley
Poem stanza 28 written by the Kurzweil Cybernetic Poet
after reading poems by William Carlos Williams