Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FR] - CLI improvements for formatting and consistency #3213

Open
disassembler opened this issue Sep 17, 2021 · 18 comments
Open

[FR] - CLI improvements for formatting and consistency #3213

disassembler opened this issue Sep 17, 2021 · 18 comments
Assignees
Labels
API&CLI-Backlog comp: cardano-api enhancement New feature or request type: enhancement An improvement on the existing functionality user type: internal Created by an IOG employee

Comments

@disassembler
Copy link
Contributor

disassembler commented Sep 17, 2021

Internal/External
Internal if an IOHK staff member.

Area

Other Any other topic (Delegation, Ranking, ...).

Describe the feature you'd like
Currently, we abuse a lot of the UI output of CLI in internal and community tools. We also trick the CLI into giving us json for some commands but not others by setting an --out-file /dev/stdout. Ideally, we'd have a unified interface that could be used for scripts for all CLI commands that provide output to the terminal. My thoughts are add a --format flag that is required for any command in CLI that gives output and initially only support plain and json gives a machine readable format that can be used for scripts.

Some examples:
for submit, something like:

{ "status": "submitted", "txid": TXID }

OR for failure

{ "status": "error", "txid": TXID, "message": "ERROR MESSAGE" }

For things like query utxo, --out-file would give the default format for the command and be overridable with the --format flag.

I think every query command absolutely needs a --format json flag added to it.

For commands that are already json, they should support a --format plain option (for example with tip.

transaction view should have the yaml format parsed as json with --format json flag.

Also, all query commands should support --out-file to put output in the specified format output to a file instead of stdout.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context / screenshots
Add any other context or screenshots about the feature request here.

@disassembler disassembler added the enhancement New feature or request label Sep 17, 2021
@AndrewWestberg
Copy link

I can't upvote this issue enough.

@gitmachtl
Copy link
Contributor

The query utxo normal output should stay as the normal output, adding a --json flag to get the output in json format makes sense. But having the normal output gives some advantages in post processing also with the number overflow limitations of some jq versions. So please keep the normal output too, or invent a --non-json flag

@disassembler disassembler changed the title [FR] - All commands with output should support --json flag [FR] - CLI improvements for formatting and consistency Dec 17, 2021
@robinboening
Copy link

I was thinking of creating such issue myself, but now I found this. This would be so helpful, thank you!

I am currently writing a cardano-cli wrapper in Ruby and its a pain not knowing what output format to expect unless you know exactly what command returns what.

One example: Splitting the query utxo output for a machine readable data is so laborious!

Than you for considering this!

@gitmachtl
Copy link
Contributor

gitmachtl commented Jan 19, 2022

This post above is a bit old. When you set the output flag it will output json.

Processing the plain text output - which will stay and is important as alternative - is not difficult. I generate my own JSON version of an UTXO with stringified numbers like: https://github.com/gitmachtl/scripts/blob/7b59fa3a41aad421d840a89fe21f14e25c36b9c3/cardano/mainnet/00_common.sh#L416-L479

The nice think about the plaintext output is to manipulate it superfast with the normal tools, filter out utxos, etc. 😄

@robinboening
Copy link

robinboening commented Jan 19, 2022

This post above is a bit old. When you set the output flag it will output json.

Could you show an example for the utxo query? I don't see it documented, nor can I get it working with the latest cardano-cli version: ​cardano-cli 1.33.0 returns Invalid option --output or Invalid option --json

Processing the plain text output - which will stay and is important as alternative - is not difficult. I generate my own JSON version of an UTXO with stringified numbers like: https://github.com/gitmachtl/scripts/blob/7b59fa3a41aad421d840a89fe21f14e25c36b9c3/cardano/mainnet/00_common.sh#L416-L479

Parsing isn't that difficult, but there is a good reason there are standardised formats like JSON. I don't think you're trying to say everyone should write their own parser as we all do at the moment, do you? I've done that myself as well, and I hate it.

Only a tiny change in the current format and your hardly readable 60 lines of code will break. I mean, c'mon you've added comments everywhere to explain why you're doing what you're doing. This is completely superfluous when choosing a standard that can be parsed by existing parsers.

The nice think about the plaintext output is to manipulate it superfast with the normal tools, filter out utxos, etc. 😄

I don't agree. There is no need to manipulate the pretty plaintext output if there is a JSON version as you can easily build any output you want, superfast, with normal tools. It's still a CLI and not a GUI tool.

@robinboening
Copy link

your hardly readable 60 lines of code will break. I mean, c'mon you've added comments everywhere to explain why you're doing what you're doing.

After re-reading my comment I thought I should come back and say I didn't mean to offend you. My comment might read harsh, but what I really mean is that any parser for this output will not be very easy to read and it should simply not be needed at all.

@gitmachtl
Copy link
Contributor

gitmachtl commented Jan 19, 2022

The CLI will output a JSON structure if there is an --out-file parameter. So a simple --out-file /dev/stdout will give you a JSON format output.

The reason why i am using the plain text output and converting it to JSON by myself to the exact same output format as the above method will give you is, that the jq tool has a problem with large numbers. The numbers of the assets are numbers and not strings, for very large ones - and we have those in cardano - the jq tool (which is the most common used on the cli) is failing. Thats the reason i redo it and i put the numbers into strings. Because the jq -r command is not having a problem with that. There is also a new one, that is much quicker than the old one. 😄
https://github.com/gitmachtl/scripts/blob/7b59fa3a41aad421d840a89fe21f14e25c36b9c3/cardano/testnet/00_common.sh#L481-L571

You can directly use the JSON format output of the query, but be careful with the calculations and do a double check on large numbers.

@robinboening
Copy link

The CLI will output a JSON structure if there is an --out-file parameter. So a simple --out-file /dev/stdout will give you a JSON format output.

Thank you!

the jq tool has a problem with large numbers

Right, that's actually a known issue for quite some years. The PR that supposedly fixes that issue got merged in 2019, but jq hasn't been released since 2018.

Maybe one day, when package managers are equipped with a jq version including that fix, we can get rid of the table representation :)

Anyways, makes sense now. Thanks for clarifying!

@cblp
Copy link
Contributor

cblp commented Jan 28, 2022

I'm working on this in #3548. Please comment here or there which commands need JSON output and which ones have wrong JSON behaviour and must be fixed.

@cblp cblp mentioned this issue Jan 28, 2022
3 tasks
@gitmachtl
Copy link
Contributor

gitmachtl commented Jan 31, 2022

i think the queryutxo is the prominent one. the default with the standard output is ok and should be kept because of the jq bug with large numbers. but an additional json flag would be nice yes.

have to check, but i think the fee calculation output is also in the normal format which could get an upgrade with an additional json flag

@CarlosLopezDeLara
Copy link
Contributor

wen json =)

@gitmachtl
Copy link
Contributor

wen json =)

what do you need in json?

@CarlosLopezDeLara
Copy link
Contributor

what do you need in json?

it would be good to have an option for this output as json

% cardano-cli query utxo --address addr1q8qu636pwlt6u4wjtnwql5ehr8vfrw9jjxstuuztsp7q08dn3uchslyht5x097hm8ghx005uxakmyuj93jk34zpsjgassp7ws9 --mainnet
                           TxHash                                 TxIx        Amount
--------------------------------------------------------------------------------------
c4892e9a0db440a878e465de5efa8ac93661fbdbf27a1f5df9496c12fd4cc2ab     0        3056593745 lovelace + TxOutDatumNone

@gitmachtl
Copy link
Contributor

gitmachtl commented Feb 24, 2022

what do you need in json?

it would be good to have an option for this output as json

% cardano-cli query utxo --address addr1q8qu636pwlt6u4wjtnwql5ehr8vfrw9jjxstuuztsp7q08dn3uchslyht5x097hm8ghx005uxakmyuj93jk34zpsjgassp7ws9 --mainnet
                           TxHash                                 TxIx        Amount
--------------------------------------------------------------------------------------
c4892e9a0db440a878e465de5efa8ac93661fbdbf27a1f5df9496c12fd4cc2ab     0        3056593745 lovelace + TxOutDatumNone

Already available, please scroll a bit up... but in short:

The CLI will output a JSON structure if there is an --out-file parameter. So a simple --out-file /dev/stdout will give you a JSON format output.

The standard output will stay, because it is handy for many applications, also we have the "big number" bug in jq, and huge numbers of NativeAssets, this breaks with the json output. If you don't use jq for further processing, you don't have to worry and you can go ahead and use the json output like described above.

@cblp
Copy link
Contributor

cblp commented May 5, 2022

What I plan to do:

  • query utxo
    • document that --out-file writes JSON
    • add --json
  • transaction view
    • add --json
  • transaction calculate-min-fee
    • add --json

@disassembler and others, please confirm this is what you expect

@gitmachtl
Copy link
Contributor

sounds good, but please don't remove the normal output on the query utxo to have tools in place to overcome the "large number" in json issues. 🙏

@cblp
Copy link
Contributor

cblp commented May 6, 2022

I don't consider it normal :), but okay, nothing will be removed (for now), only extension is planned.

@dorin100 dorin100 added type: enhancement An improvement on the existing functionality user type: internal Created by an IOG employee comp: cardano-api labels Oct 24, 2022
@Jimbo4350
Copy link
Contributor

Closing this. If this is still relevant please reopen.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
API&CLI-Backlog comp: cardano-api enhancement New feature or request type: enhancement An improvement on the existing functionality user type: internal Created by an IOG employee
Projects
No open projects
Development

Successfully merging a pull request may close this issue.

8 participants