قالب وردپرس درنا توس
Home / Tips and Tricks / How to parse JSON files on the Linux command line using jq

How to parse JSON files on the Linux command line using jq



  A terminal prompt on a Linux PC.
Fatmawati Achmad Zaenuri / Shutterstock

JSON is one of the most popular formats for the transmission of text-based data on the Internet. It is everywhere and you will find it for sure. We'll show you how to use the Linux command line with the jq command.

JSON and jq

JSON stands for JavaScript Object Notation. It is a scheme that allows data to be encoded in plain text files in a self-descriptive manner. A JSON file contains no comments. The content should be self-explanatory. Each data value has a text string called a "name" or "key". Here you can find out how high the data value is. Together they are called Name: Value Pairs or Key: Value Pairs. A colon (: ) separates a key from its value.

An "object" is a collection of key-value pairs. In a JSON file, an object begins with an open curly bracket ( {) and ends with a closing bracket (} ). JSON also supports "arrays", ordered lists of values. An array begins with an opening bracket ( [) and ends with a closing bracket (] ).

Of course, any complexity can arise from these simple definitions. For example, objects can be nested within objects. Objects can contain arrays, and arrays can also contain objects. All of this can have open levels of nesting.

In practice, however, the design of the data layout should probably be reconsidered if the layout of JSON data is complicated. If you don't generate the JSON data, but only try to use it, you obviously have no say in the layout. In these cases, you only have to worry about it.

Most programming languages ​​have libraries or modules with which they can analyze JSON data. Unfortunately, the bash shell has no such functionality.

Necessity to be the mother of the invention, although the utility jq was born! With jq we can easily analyze JSON in the bash shell. And it doesn't matter whether you have to work with sophisticated, elegant JSON or out of nightmares.

How to Install jq

We had to install jq on all of the Linux distributions with which we examined this article.

To install jq on Ubuntu, enter the following command:

  sudo apt-get install jq 

  The

How to install “sudo apt-get install jq” command in a terminal window.” width=”646″ height=”58″ src=”/pagespeed_static/1.JiBnMqyl6S.gif” onload=”pagespeed.lazyLoadImages.loadIfVisibleAndMaybeBeacon(this);” onerror=”this.onerror=null;pagespeed.lazyLoadImages.loadIfVisibleAndMaybeBeacon(this);”/>

jq On Fedora, enter this command:

  sudo dnf install jq 

  Die

To install jq on Manjaro, enter this command:

  sudo pacman -Sy jq 

  The

How to Make JSON Readable

JSON doesn't care about spaces, and the layout doesn't affect it. As long as it follows the rules of JSON grammar, systems that process JSON can read and understand it. For this reason, JSON is often transmitted as a simple, long string, regardless of the layout. This saves some space because tabs, spaces and line breaks do not have to be contained in JSON. The disadvantage of all of this, of course, is when a person tries to read it.

Let's pull a short JSON object from the NASA site that tells us the location of the International Space Station. We use curl to download files to get the JSON object for us.

We don't care about the status messages that curl normally generates Enter the following using the -s (mute) option:

  curl -s http: // api.open-notify.org/iss-now.json[19659027‹The"curl-shttp://apiopen-notifyorg/iss-nowjson"commandinaterminalwindow"width="646"height="97"src="/pagespeed_static/1JiBnMqyl6Sgif"onload="pagespeedlazyLoadImagesloadIfVisibleAndMaybeBeacon(this);"onerror="thisonerror=null;pagespeedlazyLoadImagesloadIfVBibleeaconMay/>

Now you can read this with a little effort. You have to select the data values, but this is not easy or convenient. Let's repeat this, but this time we will pass it through jq . [19659006] jq uses filters to parse JSON, and the simplest of these filters is a period (). ), which means "print the entire object" uck out jq the edition pretty.

We put everything together and type the following:

  curl -s http://api.open-notify.org/iss-now .json | jq. 

  The

This is much better! Now we can see exactly what's going on.

The entire object is enclosed in curly brackets. It contains two keys: Name pairs: message and time stamp . It also contains an object called iss_position that contains two keys: pairs of values: longitude and latitude .

We will try this again. This time we type the following and redirect the output to a file called "iss.json":

  curl -s http://api.open-notify.org/iss-now.json | jq. > iss.json 
  cat iss.json 

  The commands iss.json "and" cat iss.json "in a terminal window." width = "646" height = "262" src = "/ pagespeed_static / 1.JiBnMqyl6S.gif" onload = "pagespeed.lazyLoadImages.loadIfVisibleAndMaybeBeacon (this);" onerror = "this.onerror = null; pagespeed.lazyLoadImages.loadIfVisibleAndMaybeBeacon (this);" />

This gives us a clear copy of the JSON object on our hard drive.

RELATED:

19659041] Using Curl to Download Files from the Linux Command Line

Accessing Data Values ​​

As we have seen above jq Extract data values ​​that are forwarded by JSON. It can also work with JSON stored in a file. We will work with local files so that the command line is not crowded with curl commands. This should make tracking a little easier.

The easiest way to extract data from a JSON file is to provide a key name to get the data value. Enter a period and the key name without spaces. This creates a filter from the key name. We also need to tell jq which JSON file to use.

We type the following to get the value of the message :

  jq .message iss.json 

jq prints the text of the message ] in the terminal window.

If you have a key name that contains spaces or punctuation, you must break its filter in quotation marks. Care is usually taken to use only characters, numbers and underscores so that the JSON key names are not problematic.

First, we enter the following to get the timestamp value :

  jq .timestamp iss .json 

  The

The timestamp value is retrieved and printed in the terminal window.

But how can we access the values ​​in the object iss_position ? We can use the JSON dot notation. We will include the object name iss_position in the "path" to the key value. For this purpose, the name of the object in which the key is located precedes the name of the key.

We enter the following, including the key name for latitude (note that there are no spaces between “. Iss_position” and “.latitude”):

  jq .iss_position.latitude iss.json [19659060] The "jq .iss_position.latitude iss.json" command in a terminal window." width="646" height="97" src="/pagespeed_static/1.JiBnMqyl6S.gif" onload="pagespeed.lazyLoadImages.loadIfVisibleAndMaybeBeacon(this);" onerror="this.onerror=null;pagespeed.lazyLoadImages.loadIfVisibleAndMaybeBeacon(this);"/> 

To extract multiple values, you must do the following:

  • List the key names on the command line. Separate them with commas ().
  • Enclose them in quotation marks ( ") or apostrophes ( & # 39; ).

In this sense, we enter the following:

  jq". iss_position.latitude, .timestamp "iss.json 

  The

The two values ​​are printed in the terminal window. [19659011] Working with arrays

Let's take a different JSON object than NASA. [19659006] This time we are using a list of the astronauts currently in space:

  curl -s http: // api. Open-notif y.org/astros.json[19659070‹The"curl-shttp://apiopen-notifyorg/astrosjson"commandinaterminalwindow"width="646"height="147"src="/pagespeed_static/1JiBnMqyl6Sgif"onload="pagespeedlazyLoadImagesloadIfVisibleAndMaybeBeacon(this);"onerror="thisonerror=null;pagespeedlazyLoadImagesloadIfVisibleAndMaybeBeacon(this);"/>

Okay, that worked, so we'll do it again.

We enter the following to route it through jq and redirect it to a file called "astro.json":

  curl -s http://api.open-notify.org /astros.json | jq. > astro.json 

  The command astros.json "in a terminal window". width = "646" height = "77" src = "/ pagespeed_static / 1.JiBnMqyl6S.gif" onload = "pagespeed.lazyLoadImages.loadIfVisibleAndMaybeBeacon (this);" onerror = "this.onerror = null; pagespeed.lazyLoadImages.loadIfVisibleAndMaybeBeacon (this);" />

Now type the following to check our file:

  less astro.json 

  The [19659006] As shown below, we now see the list of astronauts in space and their spaceships.

 Edition of

This JSON object contains an array called people . We know that it is an array because the bracket opens ( [) (highlighted in the screenshot above). It is a series of objects, each containing two key-value pairs: Name and Handwerk .

As before, we can use JSON dot notation to access the values. We also need to include the parentheses ([]) in the array name.

In this sense, we enter the following:

  jq ".people [] .name" astro.json [19659085] The "jq" .people [] .name "astros.json" command in a terminal window. "width =" 646 "height =" 197 "src =" / pagespeed_static / 1.JiBnMqyl6S.gif "onload =" pagespeed.lazyLoadImages.loadIfVisibleAndMaybeBeacon (this); "onerror =" this.onerror = null; pagespeed.lazyLoadImages .loadIfVisibleAndMaybeBeacon (this); "/> 

This time all name values ​​are printed in the terminal window. We asked jq to print the name value for every object in the array. Pretty neat, right?

We can get the name of a single object by inserting its position in the array in the parentheses ([]) on the command line, the array uses zero offset, which means the object at the first position in the array is zero.

Um to access the last object in the array, you can use -1 to get the penultimate object in the array lten, you can use -2 etc.

Sometimes the JSON object specifies the number of elements in the array, which it does. Along with the array, it contains a key-name pair called number with a value of six.

The following number of objects are in this array:

  jq ".people [1]. Name" astro.json 
  jq ".people [3] .name" astro.json 
  jq ".people [-1] .name" astro.json 
  jq ".people [-2] .name" astro.json 

  Das

You can also create a start and end object within the array provide. This is called "cutting" and can be a little confusing. Remember that the array uses zero offset.

To retrieve (but not include) the objects from index two to the object at index four, enter the following command:

  jq ".people [2:4]" astro.json 

  The

This prints the objects at array index two (the third object in the array) and three (the fourth object in the array). Processing stops at array index 4, the fifth object in the array.

To understand this better, you need to experiment on the command line. You will soon see how it works.

Using pipes with filters

You can route the output from one filter to another and you don't have to learn a new symbol. Like the Linux command line, jq uses the vertical bar ( | ) to represent a pipe.

We will notify jq to direct the pipe people array into the filter .name which should list the names of the astronauts in the terminal window.

We enter the following:

  jq ".people [] | .name" astro.json 

  The

RELATED: Using Pipes under Linux

Creating Arrays and Modifying Results

We can use jq to create new objects such as arrays. In this example, we extract three values ​​and create a new array that contains these values. Note that the opening ( [) and closing parentheses (] ) are also the first and last characters in the filter string.

We type the following:

  jq "[.iss-position.latitude, iss_position.longitude, .timestamp]" iss.json 

  The

The output is enclosed in parentheses and separated by commas, making it a correctly shaped array.

Numerical values ​​can also be edited when retrieved. Let's pull the timestamp from the ISS position file, extract it again and change the returned value. To do this, type the following:

  jq ".timestamp" iss. json 
  jq ".timestamp - 1570000000" iss.json 

  The

This is useful when you need to add or remove a standard offset from an array of values.

Enter the following to remind us what the iss.json file contains:

  jq. iss.json 

  The

Let's say we want to get rid of the key-value pair . It has nothing to do with the position of the International Space Station. It is just a flag that indicates that the location was successfully retrieved. If it meets the requirements, we can do without it. (You can also just ignore it.)

We can use the jq del () delete function to delete a key-value pair. To delete the message key: value pair, enter this command:

  jq "del (.message)" iss.json 

  Das

Note that this is not actually from the " iss "will delete .json" file; it is only removed from the output of the command. If you need to create a new file without the key-value pair run the command and redirect the output to a new file.

More complicated JSON objects

Let's get some more NASA data. This time we're using a JSON object that contains information about meteorite impact sites from around the world. This is a larger file with a much more complicated JSON structure than the ones discussed earlier.

First, we type the following to redirect them to a file called "strike.json":

  curl - s https://data.nasa.gov/resource/y77d-th95.json | jq. > strike.json 

  The command strike.json "in a terminal window". width = "646" height = "77" src = "/ pagespeed_static / 1.JiBnMqyl6S.gif" onload = "pagespeed.lazyLoadImages.loadIfVisibleAndMaybeBeacon (this);" onerror = "this.onerror = null; pagespeed.lazyLoadImages.loadIfVisibleAndMaybeBeacon (this);" />

To see what JSON looks like, we type the following:

  less hits. Json 

  The

As shown below, the file begins with an opening bracket ( [) so that the entire object is an array. The objects in the array are collections of key-value pairs, and there is a nested object called Geolocation . The object Geolocation contains further key-value pairs and an array with the names coordinates .

 edition of the

let's get the name of the meteor beats from the object at index 995 through the end of the array.

We enter the following to route the JSON through three filters:

  jq ". [995:] |. [] | .name" strike.json 

<img class = "alignnone wp-image -532668 size-full "data-pagespeed-lazy-src =" https://www.howtogeek.com/wp-content/uploads/2020/01/x26-2.png.pagespeed.gp+jp+jw+pj + ws + js + rj + rp + rw + ri + cp + md.ic.mMOruXD94D.png "alt =" The filters work in the following way:

  • . [995:]: This indicates jq to process the objects from the array index 995 until the end of the array, no number after the colon (: ) says jq until the end of the array.
  • . []: This array iterator instructs jq to process each object in the array.
  • .name : This filter extracts the name value.

With a slight change we can extract the l set 10 objects from the array. A "-10" tells jq to process objects 10 back from the end of the array.

We enter the following:

  jq ". [-10:] |. [] |. Name" strike.json 

 The

As in the previous examples, we can do the following Enter to select a single object:

  jq ". [650] .name" strike.json [19659158] The "jq". [650] .name "strikes.json" command in a terminal window. "width = "646" height = "97" src = "/ pagespeed_static / 1.JiBnMqyl6S.gif" onload = "pagespeed.lazyLoadImages.loadIfVisibleAndMaybeBeacon (this);" onerror = "this.onerror = null; pagespeed.lazyLoadImages.loadMfbeiseaAnd (this) ; "/> 

We can also use string cutting by typing the following to request the first four characters of the object name in array index 234:

  jq". [234] .name [0:4] "strike.json 

 The

We can also see a specific object in its entirety. To do this, we add the following and add an array index without a key : Value filter:

  jq ". [234] "strike.json 

 The

If you only want to display the values ​​you can do the same without the key names.

In our example we enter this command:

  jq ". [234][] "strike.json 

 The

To get multiple values ​​from each object, we separate them with commas in the following command:

  jq". [450:455] |. [] | .name, .mass "strike.json 

 The

If you want to get nested values, you need to identify the objects that form the" path "to them.

For example, to To reference the values ​​of the coordinates you must include the comprehensive array, the geolocation nested object, and the nested coordinates array as shown below.

 The path to the array in a nested JSON object, highlighted in a terminal window.

To display the coordinate values ​​ for the object at index 121 of the array, issue the following command a:

  jq ". [121]. Geolocation. Coordinates [] "strike.json 

 The

The Length Function

The jq Length Function gives different metrics depending on what was applied such as

  • Strings : The length of the string in bytes.
  • Objects : The number of keys: value pairs in the object.
  • Arrays : The number of array elements in the array.

The following command returns the length of the value name in 10 objects in the JSON array from index position 100:

  jq ". [100:110] | , [] .name | Length "strike.json 

 The

To see how many key-value pairs are in the first object of the array, enter this command:

  jq". [0] | Length "strike.json 

 Die

The key function

With the key function you can find out about the JSON you have to work with. You can give them the names of the keys and how many objects are in an array.

To find the keys in the object people in the file "astro.json", enter the following command:

  jq ". People. [0] | Key "astro.json 

 The

To see how many elements are in the array people enter this command:

  jq" .people | Keys "astro.json 

 Die

This shows that there are six zero offset array elements numbered from zero to five.

The has () function

You can use The function has () queries the JSON and checks whether an object has a specific key name. Note that the key name must be enclosed in quotation marks. The filter command is placed in single quotation marks ( & # 39; [19459010)]) as follows:

  jq & # 39 ;. [] | has ("nametype") & # 39; strike.json 

 The

Each object in the array is checked as shown below. [19659006]   Output from "jq". [] | has ("nametype") the command & # 39; strike.json "in a terminal window.

To review a specific object, add its index position to the array filter as follows:

  jq & # 39; [678] | has ("nametype") & # 39; strike.json 

 The

Don't go near JSON without it

The utility jq is that perfect example of this The professional, powerful and fast software that makes life in the Linux world a pleasure.

This was just a brief introduction to the general functions of this command - there's a lot more to it than this if you want to dig deeper.




Source link