0

I've seen many SO questions around auto-generating unit tests, or documenting unit tests, but I'd like to go the reverse direction. What I'd REALLY like is to have folks (devs, tech writers, etc) write documentation with embedded code, then, as part of CI, automatically convert that documentation to runnable code, and pass/fail the pipeline based on the ability to run. The goals here would be to:

  1. Have more documentation driven development (where the 'T' in TDD is a test derived from the document)
  2. Ensure that the documentation is always up to date (because if it fails, either the code or the documentation has to be updated to pass)
  3. Encourage more writing of tests, particularly by folks designed to represent the users (PMs, support, etc).

So, specifically, I would like to go from something like the following (doc.md):

To use this feature, first you need to create an environment:

`
cli create environment
`

Then you need to execute the program:

`
cli program execute --arg fooz
`

This will result in the program being executed on the server.

So the idea is to go from this to an automated test that ran:

cli create environment
cli program execute --arg fooz

assert stdout == "success"

This is a pretty trivial example, but we shouldn't need more than that. We could certainly write it ourselves, but this seems like the kind of thing that would already be out there.

A more complete example that would be converted automatically and run during CI would be something like this -

https://cloud.google.com/deep-learning-vm/docs/quickstart-cli

I'm not looking for BDD or other tests that do not convert into user facing documentation, or auto generated anything (as you can see the documentation above would have to be hand written, ideally by the developer).

Improve this question
10
  • 3
    Didn't downvote, but this is hard to answer without a specific language in mind. Python something similar to his in the form of doctest. Also, I think your question is sending mixed signals: "I've seen many SO questions around auto-generating unit tests ... but I'd like to go the reverse direction" and then later "where the 'T' in TDD is a test derived from the document". So which way do you want to go, form tests to docs, or docs to test? Commented Oct 4, 2021 at 23:51
  • Sorry about that! I've tried to flesh it out a bit more Commented Oct 5, 2021 at 2:45
  • 3
    This sounds like BDD, which is often automated using tools such as Cucumber - cucumber.io Commented Oct 5, 2021 at 10:05
  • I might be getting too hung up on your trivial example, but wouldn't an install script do the same thing? Commented Oct 5, 2021 at 11:22
  • The issue with cucumber is it's just BDD. This means writing things which are useful for developers, but not for users. Commented Oct 5, 2021 at 12:30

2 Answers 2

Reset to default
2

There are some tools around that come close to what you want (FitNesse comes to mind). You still have to write a lot of glue code that connects the facts and conditions from the documentation to running code. It boils down to how "smart" the language of the documentation and tooling is, if you have to write a lot of additional code.

See it like this: Code is all about details and test code is no exception from this. You need to express these details in the documentation to enable it to call the code that you wrote. With plain english you only get a part of the way to tests derived automatically from documentation. But there is a language that can provide all the details that you need to generate tests that call other code. It is called a programming language.

There have been uncountable attempts to create a programming system, that the business folks can use to write their application without us pesky progammers. They failed every time. Why? Because of the details.

Improve this answer
1
  • Fitnesse looks interesting! To be clear, I'm definitely not looking to have business people write intents - the stuff in the miiddle of the works is fully executable code. But going beyond just regex deleting non-code to additional functionality (eg capturing intents, hidden code sections, etc) would be really powerful Commented Oct 5, 2021 at 8:19
0

Sounds like BDD, Gherkin, Cucumber etc. Or these "no-code" style Business Rules Engines The problem is can you get product owners to write and care about the tests/specs/docs.

In theory it works because the customer sits with you a in a meeting and agrees these plain English specifications, but in practice that's not what happens.

Customers want untestable things like "Like facebook but with more buttons" and get frustrated if you try and get them to commit to details. To them it feels like you are trying to trick them into missing out key features so they can be blamed when it doesn't work.

If if you get one who understands and agrees to a more details spec, they may not care if that spec changes slightly due to other constraints or conflicting requirements. Your test may fail, but its OK because the intent was satisfied and you ran out of time, or you got a verbal OK in some meeting that the button should be blue to match the style rather than red as originally specced etc.

For the last 20+ years developers have been trying to move away from the waterfall approach of "get a perfect spec before you start coding" That approach makes the coding easy, but makes writing the spec a programming task!

Similarly with documentation If you want code to be automatically generated from documentation, the complexity of the documentation increases 100 fold. Writing the documentation is no longer "Write human readable instructions on how to achieve common tasks" but "Write the code in a new programming language which makes it read like documentation".

Same with the BRE, Sure the drag n drop interface can be used by Mr Mid-Level-Manager when its the simple thing in the demo the sales man shows you, "Order Save to Database AND Emails to Accounts!!" but when its "decode the oauth claims, and then recurse the tree to find the reporting contact, oh but remember to check you don't go over max int and the business year ends on the first full moon of October" you need a team of devs to wrangle the spaghetti mess the users have organically created in year 1

Improve this answer
7
  • Hi! Thank you for the response - I'm looking for something sightly different. Definitely not looking for complete spec, or universal auto generated docs, more like documented tutorials that become part of your unit tests. For example, a tutorial like this - cloud.google.com/deep-learning-vm/docs/quickstart-cli - this could be converted automatically into a test, and would fail if the API signature changed. Commented Oct 5, 2021 at 12:36
  • imagine you have just written that document. To make it a test, you have to populate all the $Instance_name and other variables mentioned in the example commands. To do that you need output from the other docs you linked to such as "setting up an account" you will need to transform that output and link it correctly, you will need to link your instructions for which instance type to use to the inputs of that link. But you cant add extra behind the scenes code. the document itself has to have keywords or conventions such that all these things are defined just by typing the words you see Commented Oct 5, 2021 at 12:51
  • further more, you have to have the tests computable. "The delete command deletes Compute Engine instances." somehow translates to "after running the above command, list the instances and verify that the one you created earlier (or in some setup??) doesn't exist anymore. Even if you could get it to somehow work, think about how it restricts the language you can use in the help doc. I cant say "the delete command will delete.." or "this will lead to..." maybe i cant even leave off the full stop, can I say instance instead of instances? or Instance? Commented Oct 5, 2021 at 13:08
  • I mean - yes, much of what you said is correct. But the script is not autogenerated, it's the actual script inserted in the document by the developer. I can tell you first hand that we did this, on this very document and thousands more - it was an internal tool and it was fantastic. I was hoping there was something that do this that didn't require being employed by Google. Commented Oct 5, 2021 at 13:45
  • And to handle the behind the scenes code, there WAS behind the scenes code, also written by the dev, and wrapped in instructions so that during publication it was stripped, but during the CI test, it ran. Commented Oct 5, 2021 at 13:48

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.