Natural language processing to identify grammar in a text?

Sadly, this doesn't exist yet.

You will find that most Natural Language Processing (NLP) tools conceptualize linguistic categories differently from how teachers do (language teaching isn't linguistics, there are often simplifications happening, and schoolbooks get updated more slowly than linguistics evolves).

Examples:

* English verbs have only two tenses: PAST or NONPAST. They can have PERFECTIVE aspect or not. They can have PROGRESSIVE aspect or not. Since these are 3 binary choices, there are at least 8 different ways how English verbs can be realized. I think there'd be less confusion in school if a more linguistically correct version was taught that separates out tense and aspects.

* "Future" or "Present Perfect" (something I still got taught in school) don't exist for a proper linguist.

To build what you suggest, existing tools could be combined, but there would have to be a mapping layer on top of syntactic parsers like Charniak Parser, Collins parser or MaltParser. Another mis-match between grammar in school and linguistics is single versus multiple theories: in school, people usually teach constituent trees, whereas in linguistics phrase structure (constituent) grammar is one theory among many, one alternative (valency and dependency grammar) that does not rely on trees but focuses on the relations between words has recently gained a lot of traction in linguistic circles.

I don't think there's an out-of-the box library for things like the detection of passive voice or indirect questions. But you should be able to build something that would let you do it based on the basic NLP toolkit: dependency parsing, POS tagging, lemmatization, named entity recognition.

I suggest you check out Spacy [0] for a quick and easy to use Python library providing the above features. The software produced by the Stanford NLP Group is also great [1].

If you do not want to get your hands dirty with code, there are a number of API providers that will offer you the same as the above libraries (TextRazor, Rosette Text Analytics...)

[0] https://spacy.io/

[1] https://nlp.stanford.edu/software/

There are several topics intertwined with solutions you seek: There's parts of speech (PoS) tagging, reducing to Lemma form, identifying end of sentence, etc.

After having faced a similar learning curve, I put what I know into a lengthy document[0] written in 2018 based upon explorations over 2016-17. That will get you deployed and operational quickly by following just the final section. The first section explains key concepts using conventional ideas as the means of introducing NLP jargon. In between covers theory and practice for getting the most out of any tool you're likely to use in the end.

More general tools are probably available today, such as add-ons for Elasticsearch. I'd start looking there. Interesting items came up when searching ddg for: NLP elasticsearch.

[0] http://play.org/articles/introduction-to-natural-language-pr...

I actually (sort of) wrote one of these a while back (though I don't think I ever got to implementing tenses - possibly this would be somewhat easy to implement on top of whatever I already built there, but maybe not idk). In any case:

> copulae verbs, linking verbs, terms that are often filtered (i.e. stop terms), question terms, time sensitive nouns, amplifiers, clauses, coordinating conjunctions, negations, conditionals (ORs), and contractions

https://github.com/nyxtom/salient/

As someone else alluded to, this is a task for multiple models. Fortunately, there are a lot of great NLP libraries that combine multiple pre-trained language models into a single pipeline you can interface with, like Stanza. From their docs, their vanilla pipeline breaks down the sentence "Barack Obama was born in Hawaii. He was elected president in 2008." as :

('Barack', '4', 'nsubj:pass')

('Obama', '1', 'flat')

('was', '4', 'aux:pass')

('born', '0', 'root')

('in', '6', 'case')

('Hawaii', '4', 'obl')

('.', '4', 'punct')

It should be very easy to deploy Stanza's pipeline as an API endpoint. Here is an example of such a NLP-library-as-API endpoint, albeit with Hugging Face's Transformers, deployed via Cortex: https://github.com/cortexlabs/cortex/blob/master/examples/py...

Example of passive voice detection using spacy[1]: https://gist.github.com/armsp/30c2c1e19a0f1660944303cf079f83...

[1] https://spacy.io/

My IOCCC entry [0] detects English passive constructions. Do not feel discouraged by its looks; it is a solid tool. Since both ioccc.org and its official mirrors (in the same domain) are down now, you can look at its Wayback Machine cache [1].

[0] https://www.ioccc.org/2018/ciura/ [1] https://web.archive.org/web/20200224040340/https://www.ioccc...

Some parser like Spacy can give some tense additional information for verbs. But it's probably not custom enough for what you want.

Maybe you can give GPT-3 a try.

If you want to go the custom route, the easy way but which consume a lot of processing power, is to use a neural network and necessitate a boring dataset phase construction.

You build a dataset corresponding to your problem. And you learn it with the neural network.

For inspiration you can see my colorify browser extension, which uses a neural network to learn at the same time to split sentences, predict POS tags, predict root of the sentence, predict the parse tree which are then used to decorate the webpage.

What I did was just programmatically build a dataset from the spacy parser to build a custom javascript parser which does what I want. If I wanted to add some additional information that Spacy doesn't provide like grammar tenses and features, I can complete my dataset manually and have the network predict all the decorations at the same time, which allows it to not need a lot of samples because the layers are shared.

You can probably build your dataset faster by interacting with your neural network (active learning).

For the model you can start with something simple like a convolution residual network architecture. And later use some transformers when you want to reach state of the art.

You should look into the English Resource Grammar:

http://moin.delph-in.net/ErgTop

Online demo:

http://erg.delph-in.net/logon

It has all that information in the generated feature structure -- even more than you can view in the web interface. There's a development environment you can download, as well as a headless linux tool called ACE you can use on a server. The ERG is complex, but far and away the most sophisticated tool in this space.

Check out UDify for dependency parsing with universal parts of speech and features.

https://github.com/Hyperparticle/udify

This isn't really a programming solution but you should check out this webapp:

http://www.hemingwayapp.com/

It may not be the exact thing you're looking for but it can probably be helpful to your students.

I would also look at Python NLTK. I've only dabbled in the toolkit, so I'm not sure if it has what you're looking for exactly, but it's worth a look.

http://www.nltk.org/

To do the grammar tense analysis, you can use spaCy or another syntactic parser. The parse tree won't directly give you the exact grammar tense, you will need to do some simple analysis of the conjugational form of the root verb, and the auxiliary verbs that are attached to it.

I've done extensive work in this area, including developing my own statistical parser from scratch. I'd be happy to chat more about this project, my email is daniel dot burfoot at gmail.com.

Have you taken a look at Google's Natural Language API? Try out the demo and switch to the "Syntax" tab to see the output. More info: https://cloud.google.com/natural-language

I have played around with similar projects. A good starting point is google's NLP sentence parsing API. Be warned: the accuracy may not be good enough for your application.

You can get most of the way there with Stanford CoreNlp.

I'm not sure if it has an API or any kind of integrations, but I know the Hemingway App[0] can detect passive voice, and possibly other features you're looking for.

[0]: http://www.hemingwayapp.com/

On recent debians/ubuntu, PoS tagging is just one apt away:

    $ sudo apt install -y apertium-eng

    $ echo "I have been searching for a tool that can scan a paragraph" |apertium eng-disam|grep -v '^;'
    ""
            "prpers" prn subj p1 mf sg
    ""
            "have" vbhaver inf
            "have" vbhaver pres
    ""
            "be" vbser pp
    ""
            "search# for" vblex ger SELECT:177
    ""
            "a" det ind sg
    ""
            "tool" n sg
    ""
            "that" cnjsub
            "that" prn dem mf sg
            "that" prn rel an mf sp
    ""
            "can" vbmod pres SELECT:281
    ""
            "scan" vblex inf SELECT:140
    ""
            "a" det ind sg
    ""
            "paragraph" n sg
    "<.>"
            "." sent

(grepping out lines with ; since they just show what was not removed by the disambiguator, whereas SELECT/REMOVE are just trace info saying what rules applied. If there are multiple indented lines, then the disambiguator didn't manage to fully disambiguate the analysis.)

If you want to e.g. mark passive, it's easy to write a Constraint Grammar rule to do this. Put the following into rules.cg3:

    DELIMITERS = sent ;
    ADD (&PASSIVE) ("be") # Add the tag "&PASSIVE" to the word with lemma "be"
    IF
    (1* (pp)             # There is a participle to the right
     BARRIER (*) - (adv) # with nothing in between except perhaps adverbs
    );

and pipe it in after the above pipeline:

    $ echo "The paper is not signed by me" |apertium eng-disam |grep -v '^;'|vislcg3 -g rules.cg3
    ""
            "the" det def sp
    ""
            "paper" n sg
    ""
            "be" vbser pres p3 sg &PASSIVE
    ""
            "not" adv
    ""
            "sign" vblex pp
            "sign" vblex past
            "signed" adj
    ""
            "by" pr SELECT:470
    ""
            "prpers" prn obj p1 mf sg
    "<.>"
            "." sent

( https://wiki.apertium.org/wiki/Constraint_Grammar for more info on CG )