HCL Compass Rest APIs In Action

Introduction

This article talks about RESTFUL services which comes with HCL Compass 2.0.2 version that helps to build or interact with applications based on Compass data. HCL Compass outside of the web or desktop interfaces also provides the use of Perl APIs and a Java Native Interface. These two options tended to be complicated and lacked in flexibility which made building Applications or integrations around Compass difficult and time-consuming.

The HCL Compass REST API allows developers to interact with HCL Compass by using HTTP based REST APIs. Rest API’s can be used to do many functionalities in HCL Compass including viewing or modifying records, Creating and editing queries, Doing Text Searches and many more. Rest API features are part of the HCL Compass installation as a component. This is deployed as a Spring Boot Application instance and runs independent of the CQweb service. Once installed, you just need to launch the services to start using it. In this blog we will develop a Python script which makes use of the REST API’s to interact with HCL Compass Data. While building this, we will be introduced to some of the common RESTAPI’s used.

What this article is not about

This article is not about programming or providing code to achieve certain tasks. This is merely a way to demonstrate how we can use REST API’s in a practical context to interact with the Compass Data. Of course, there are more easier and efficient ways to write Python scripts such as for eg : using functions and in many other ways. Some of the data manipulation code is not included in the snippets and it mostly deals with how to use requests modules to interact with REST API’s.

What does the script do

1. Show the list of Repositories available in the system once you run it.
2. Allows you to pick a repository from the list to authenticate to by prompting the username, password and database as Parameters.
3. List the databases associated with the Repository chosen.
4. Ask the user to provide a Database to build a query for displaying the records assigned to that user.
5. Execute the query and displays the result set.
6. Removes the query.
7. Logs out from the REST server.

API’s used in the script

1. Repos API – Retrieves the list of Schema Repositories.
2. Authenticate API – To Authenticate to CCM REST Server.
3. Databases API – To list the databases associated with a Schema Repository.
4. Folders API – To list a parent folder to create a Query.
5. QueryDefs API – To create a Query Definition.
6. ResultSets API – Executes a Query definition and builds a Result set.
7. ResultSet API – Retrieves the single page of a previously executed Query Definition.
8. QueryDef API – To delete a Query Definition.
9. Authenticate API – To log off from CCM Rest Server.

Now, let us examine these API’s in detail while building the script,

1. Retrieving the List of Schema Repositories

Our first task is to retrieve the list of Schema Repositories for the user to help them select their own. For this, we will use the REPOS API, and specifically the end point, /ccmweb/rest/repos. We will have to use a GET HTTP request for this purpose. In this demo, we will be using Python Requests module to build the REST API script. There fore we would need to import the module to the script. And we will use the requests.get() function to send this API Request. The request.get function expects the Repository end point as a url and headers to be passed on as arguments. For this specific API, the header expected is,

‘Content-Type’: “application/json”

The header information is saved into a variable, headers and url is saved into a variable, repo_endpoint. This is going to be the standard template to be followed for passing the parameters to requests module throughout this script to make a Rest API request.

cont_type = “application/json”

headers={‘Content-Type’: cont_type}

repo_endpoint = “https://servername:8190/ccmweb/rest/repos”

response = requests.get(

   repo_endpoint,

    headers=headers

)

The above code will fetch us the list of repositories available in the system.

2.Authenticating to the CCM REST server

Now that we have got our Repositories available in the system to show the user to select one to login, we need to authenticate. For authenticating in to the REST Server, we use the Authenticate API with end point, /ccmweb/rest/authenticate. The HTTP request used is POST. We first ask the user to enter, their username, password, Repository name and Database they wish to login. Unlike the previous REPOS API, the Authenticate API requires, we send the above information in the body in JSON Format. There fore other than the headers and the url, this time, we have to provide body data also in the requests.post method. We also need to know if the request raises any exception or error message, for that we use, raise_for_status() method. If all goes well, this request would fetch us the Authentication Token in the JSON format as output which we need to use for any further API requests to fetch/modify Compass data. We need to process this JSON output to be converted into Python Dictionary data structure so we can manipulate this data and obtain the token. For this, we use the .json() method. Json.dumps function converts json data into strings so that it can be processed by Python. Notice that we are passing the same headers as in the last API call this time also,

Header expected is ‘Content-Type’: “application/json”

reponame = input(“\nEnter Repository Name : “)

auth_url = “https://servername:8190/ccmweb/rest/authenticate”

username = input(“\nEnter your username : “)

password = input(“\nEnter your password : “)

db1name = input(“\nEnter db name : “)

body_data={“username”:username,

       “password”:password,

       “repo”:reponame,

       “db”:db1name

   }

auth_response = requests.post(auth_url,headers=headers,data=json.dumps(body_data))

auth_response.raise_for_status()

auth_data=auth_response.json()

3.List the Databases associated with the Schema Repository

For listing the databases associated with a schema repository we use, Databases API with endpoint, ccmweb/rest/repos/repository name/databases. The HTTP request is GET. This time apart from the headers in the previous API call, we need to send the Authentication Token (which we received as output after making the previous API call) also in the following format,

Headers expected are  ‘Content-Type’: “application/json”,

Authorization: Bearer eyJhbGciOiJIUzUxMiJ9.eyJzdWIiOiJhZG1pbiIsInJlcG8iOiIxMC4wLjAiLCJleHAiOjE1NzM2NjE5NDUsImlhdCI6MTU3MzU3NTU0NSwiZGIiOiJTQU1QTCJ9.Lgdk9gPm6mFFGEnr6mRt7Vrq-nTnlP86ARKc16bP4syeTIvKQ55JX5r6aFXVC20xC2NwsjqbvAhd2f1r2PiIUA

In the requests.get function, we pass url and headers. We also raise_for_status() and read the response with .json function. This API does not have any mandatory body data. We should get the databases as dictionary which we can process and display to the user to select for accessing the Compass data.

auth = “Bearer”+” “+auth_data[‘token’]

headers1={‘Content-Type’: cont_type,

        ‘Authorization’: auth

         }

db_url = f”https://servername:8190/ccmweb/rest/repos/{RespositoryName}/databases”

dbname_response = requests.get(db_url,

                               headers=headers1

         )

dbname_response.raise_for_status()

dbnames = dbname_response.json()

4.Get Parent Folder DBID to create a Query

We need to create a Query in some folder in the workspace. There fore we need to find out which folders are there in our root workspace. In this example, we are going to store our Query in the Public Queries folder since it is temporary. Hence, we need to find out the DBID of the Public Queries folder in our workspace. We use the Folders API for this purpose with endpoint as, /ccmweb/rest/repos/ repository name /databases/ database name /workspace/folders.  HTTP Request used is GET.

Headers expected are ‘Content-Type’: “application/json”,

Authorization: Bearer eyJhbGciOiJIUzUxMiJ9.eyJzdWIiOiJhZG1pbiIsInJlcG8iOiIxMC4wLjAiLCJleHAiOjE1NzM2NjE5NDUsImlhdCI6MTU3MzU3NTU0NSwiZGIiOiJTQU1QTCJ9.Lgdk9gPm6mFFGEnr6mRt7Vrq-nTnlP86ARKc16bP4syeTIvKQ55JX5r6aFXVC20xC2NwsjqbvAhd2f1r2PiIUA

We need to pass the parameters to the requests.get function and as usual, use raise_for_status function and catch the output response using .json function. We pass the JSON data in json.dumps function. We are interested in the dbId data from this output which will required for our next API,

parent_url = f”https://servername:8190/ccmweb/rest/repos/{RepositoryName}/databases/{DatabaseName}/workspace/folders”

parent_dbid_response = requests.get(parent_url ,headers=headers1)

parent_dbid_response.raise_for_status()

parent_dbid_higher = parent_dbid_response.json()

parent_dbid = parent_dbid_higher[1][‘dbId’]

5.Create a Query Definition

Once we get the input from the user as to which Database they need access to and Parent DBID, the purpose of our script is to display the user which defects in that database are owned by them. For this, in the first step, we need to build a Query definition to display all Defects which are owned by the Login User. We use the QueryDefs API for this purpose with endpoint as /ccmweb/rest/repos/repository name/databases/database name/workspace/queryDefs. HTTP request used is POST. This API requires the same headers like the previous API but mandatorily requires body data which forms the structure of the Query to be constructed.

Headers expected are ‘Content-Type’: “application/json”,

Authorization: Bearer eyJhbGciOiJIUzUxMiJ9.eyJzdWIiOiJhZG1pbiIsInJlcG8iOiIxMC4wLjAiLCJleHAiOjE1NzM2NjE5NDUsImlhdCI6MTU3MzU3NTU0NSwiZGIiOiJTQU1QTCJ9.Lgdk9gPm6mFFGEnr6mRt7Vrq-nTnlP86ARKc16bP4syeTIvKQ55JX5r6aFXVC20xC2NwsjqbvAhd2f1r2PiIUA

The body data is as below for our purpose which is stored into a variable query_data,

query_data={ “name”: “MyDefects”,

   “dbIdParent”: parent_dbid,

   “primaryEntityDefName”: “Defect”,

   “queryFieldDefs”: [

       {

           “fieldPathName”: “Headline”,

           “isShown”: “true”

       }

                     ],

   “filterNode”: {

       “boolOp”: “BOOL_OP_AND”,

       “fieldFilters”: [

       {

       “fieldPath”: “Owner”,

       “compOp”: “COMP_OP_EQ”,

       “values”: [username],

       “stringExpression”: “= username”

       }

                       ]

                 }

   }

Let us look at the body data in a bit more detail. As usual, body data is in JSON format.

name – Name of the query.

dbIdParent – The dbid of the parent directory where the query is to be created.

primaryEntityDefName – Record type name.

queryFieldDefs  – fieldPathName  is the field name.

isShown : true – To display the specific field in the result set.

filterNode – boolOp – AND – Operator for adding multiple filters to the query.

fieldFilters – These are the fields to provide as filters.

fieldPath : Owner – Field name of the filter

compOp – COMP_OP_EQ – Denotes Equal to

values : [username] – username to be replaced with actual username of the user, in this condition it is, field Owner is equal to username provided. So query should list all Defects which has the owner as the username provided.

Now that we have our parameters for the API ready, we need to pass these to the requests.post function and as usual, use raise_for_status function and catch the output response using .json function. We pass the JSON data in json.dumps function. We are interested in the dbId data from this output which will be required for our next API,

query_db = input(“\nEnter Database name to build query : “)

query_data={ “name”: “MyDefects”,

   “dbIdParent”: parent_dbid,

   “primaryEntityDefName”: “Defect”,

   “queryFieldDefs”: [

       {

           “fieldPathName”: “Headline”,

           “isShown”: “true”

       }

                     ],

   “filterNode”: {

       “boolOp”: “BOOL_OP_AND”,

       “fieldFilters”: [

       {

       “fieldPath”: “Owner”,

       “compOp”: “COMP_OP_EQ”,

       “values”: [username],

       “stringExpression”: “= username”

       }

                       ]

                 }

   }

query_url =  f”https://servername:8190/ccmweb/rest/repos/{RepositoryName}/databases/{DatabaseName}/workspace/queryDefs”

query_response = requests.post(query_url,headers=headers1,data=json.dumps(query_data))

query_response.raise_for_status()

query_out = query_response.json()

query_dbid = query_out[‘dbId’]

6.Execute the Query Definition and Build a Result Set

Now that we have received the Query Dbid from the previous step, we need to use it to execute the query and build a result set. Both these operations are performed by the ResultSets API. HTTP Request used is POST. We use the following end point for this purpose, /ccmweb/rest/repos/repositoryname/databases/databasename/workspace/queryDefsquerydbid/resultsets. Headers used are the same as from the previous step,

Headers expected are ‘Content-Type’: “application/json”,

Authorization: Bearer eyJhbGciOiJIUzUxMiJ9.eyJzdWIiOiJhZG1pbiIsInJlcG8iOiIxMC4wLjAiLCJleHAiOjE1NzM2NjE5NDUsImlhdCI6MTU3MzU3NTU0NSwiZGIiOiJTQU1QTCJ9.Lgdk9gPm6mFFGEnr6mRt7Vrq-nTnlP86ARKc16bP4syeTIvKQ55JX5r6aFXVC20xC2NwsjqbvAhd2f1r2PiIUA

This API does mandatorily accept the body data. If you do not have any specific parameter to pass on you can just pass an empty json like below,

query_exec_data={

    }

Other than this, url, data and headers are passed on to the requests.get function. As usual, we will use the raise_for_status, json.dumps and .json functions here also. From the output we are expecting to get “result_set_id” to pass on to the next API request.

query_exec_url = f”https://servername:8190/ccmweb/rest/repos/{RepositoryName}/databases/{DatabaseName}/workspace/queryDefs/{Querydbid}/resultsets”

query_exec_data={

   }

query_exec_response=requests.post(query_exec_url,headers=headers1,data=json.dumps(query_exec_data))

query_exec_response.raise_for_status()

query_exec_results = query_exec_response.json()

resultset_id = query_exec_results[‘result_set_id’]

7. Display the Query ResultSet to User

It’s time to display the result set to the user. For this, we use the, ResultSet API with the endpoint, /ccmweb/rest/repos/repositoryname/databases/databasename/workspace/queryDefs/querydbid/resultsets/resultsetid. We got the resultsetid as an output from previous step. HTTP Request is POST. It just requires the url and headers as parameters.

Headers expected are ‘Content-Type’: “application/json”,

Authorization: Bearer eyJhbGciOiJIUzUxMiJ9.eyJzdWIiOiJhZG1pbiIsInJlcG8iOiIxMC4wLjAiLCJleHAiOjE1NzM2NjE5NDUsImlhdCI6MTU3MzU3NTU0NSwiZGIiOiJTQU1QTCJ9.Lgdk9gPm6mFFGEnr6mRt7Vrq-nTnlP86ARKc16bP4syeTIvKQ55JX5r6aFXVC20xC2NwsjqbvAhd2f1r2PiIUA

We will use Python Prettytable package to display the results in a tabular format. The resultset output provides us in dictionary format. We get the values for rows from it and then extract the values that we need as Python Lists. Notice that in the query construct data we only requested the field, Headline in the display field, but the output contains displayName by default which is nothing but the ID of the record.

result_set_url = f”https://servername:8190/ccmweb/rest/repos/{RepositoryName}/databases/{DatabaseName}/workspace/queryDefs/{Querydbid}/resultsets/{Resultsetid}”

result_set_response = requests.get(result_set_url,headers=headers1)

result_set_response.raise_for_status()

resultset_results = result_set_response.json()

second_final = resultset_results[‘rows’]

headline_value = []

id_value = []

for i in second_final:

   headline_value.append(i[‘values’])

   id_value.append(i[‘displayName’])

x.add_column(“ID”,id_value)

x.add_column(“Headline”,headline_value)

print(x)

8.Removing the Query Definition

Our next task is to delete the Query that we created so that it is not visible in Compass database anymore as each time we run the script a new query is constructed. For this we will use the QueryDef API with the following end point, /ccmweb/rest/repos/reposittoryname/databases/databasename/workspace/queryDefs/querydbid. HTTP Request for this is DELETE. It will take url and headers as the parameters.

Headers expected are ‘Content-Type’: “application/json”,

Authorization: Bearer eyJhbGciOiJIUzUxMiJ9.eyJzdWIiOiJhZG1pbiIsInJlcG8iOiIxMC4wLjAiLCJleHAiOjE1NzM2NjE5NDUsImlhdCI6MTU3MzU3NTU0NSwiZGIiOiJTQU1QTCJ9.Lgdk9gPm6mFFGEnr6mRt7Vrq-nTnlP86ARKc16bP4syeTIvKQ55JX5r6aFXVC20xC2NwsjqbvAhd2f1r2PiIUA

Apart from the normal functions used in the previous API requests, this time the difference is, we are not going to receive any output. Hence in this case, how will we know if the query has been successfully removed? For this we rely on the HTTP status codes. If the HTTP response code is usually in the 200 range, which means our Request is successful. Hence we need the response.status_code function to retrieve the status code from the HTTP response which we can pass on to a conditional statement to verify if the Query removal is successful.

remove_query_url = f”https://servername:8190/ccmweb/rest/repos/(RepositoryName}/databases/{DatabaseName}/workspace/queryDefs/{Querydbid}”

remove_query_response = requests.delete(remove_query_url,headers=headers1)

remove_query_response.raise_for_status()

remove_query_final_response = str(remove_query_response.status_code)

if remove_query_final_response.startswith(’20’):

   print(“\n***Removing Query Successful***”)

else:

   print(“\n***Removing query failed! Please manually remove***”)

9. LogOff from the CCM REST Server

As we have successfully displayed the query output to the user, we have completed all the tasks we intended to. The only thing left is to LogOff from the server which is important to not accumulate sessions and load the server. For this purpose, we use the Authenticate API with the end point, /ccmweb/rest/authenticate/logoff. HTTP Request is POST. As like the previous API request it only takes url and headers as parameters. We will also be looking for the status code since there will not be any output to know if the logout is successful much like we did in the previous step,

Headers expected are ‘Content-Type’: “application/json”,

Authorization: Bearer eyJhbGciOiJIUzUxMiJ9.eyJzdWIiOiJhZG1pbiIsInJlcG8iOiIxMC4wLjAiLCJleHAiOjE1NzM2NjE5NDUsImlhdCI6MTU3MzU3NTU0NSwiZGIiOiJTQU1QTCJ9.Lgdk9gPm6mFFGEnr6mRt7Vrq-nTnlP86ARKc16bP4syeTIvKQ55JX5r6aFXVC20xC2NwsjqbvAhd2f1r2PiIUA

logout_url=”https://servername:8190/ccmweb/rest/authenticate/logoff”

logout_response = requests.post(logout_url,headers=headers1)

logout_response.raise_for_status()

logout_final_response = str(logout_response.status_code)

if logout_final_response.startswith(’20’):

   print(“\n***Logout Successful***”)

else:

   print(“\n***Logout Failed!***”)

Complete Output from the Script

The complete output of the script is as below. It lists us the Repositories in the System. Expects us to provide username, password and Database for Authentication to REST Server.

Then finds the parent directory to build a query to list the records owned by the user. Once after creating and displaying the records, it will remove the query and log out from the REST Server reporting along the way of any error.

***The repositories available on this server are***

———————————–

1. CadenceTest
2. RestAPIDemo

***Authenticating to Compass Rest API server!***

Enter Repository Name : RestAPIDemo

Enter your username : admin

Enter your password :

Enter db name : RESTD

***Listing the Databases for the repository RestAPIDemo ***

***The database name(s) are as follows***

RESTD

restn

***Building a Query to display records assigned to you***

Enter Database name to build query : RESTD

***Executing the Query***

***Displaying the Result Set***

+—————+———————————–+

|       ID      |              Headline             |

+—————+———————————–+

| RESTD00000041 | [‘Testing RestAPI Demo Record 1’] |

| RESTD00000042 | [‘Testing RestAPI Demo Record 2’] |

+—————+———————————–+

***Removing the Query***

***Removing Query Successful***

***Logging off***

***Logout Successful***

>>> 

Conclusion

With that, we have accomplished what we set out to do, ie to build a simple script to demonstrate the use of COMPASS HTTP REST APIs. We hope that, this will help you in getting started with developing more complex Applications and integrations with HCL Compass data.

References

https://help.hcltechsw.com/compass/2.0.2/com.hcl.compass.doc/webhelp/oxy_ex-1/com.ibm.rational.clearquest.oslc_cmrest_api.doc/topics/c_rest_api_introduction.html