Introduction

The purpose of this guide is to provide information about the available monitoring systems within the gateway, as well as list the procedures and ways of resolving the issues that may occur.

There are two audit systems in the gateway:
1) System audit - verifies whether transaction processing and system processes are executed correctly. The verification is done on daily, hourly and quarter-hourly basis. Once the issue has been identified, the system sends an email notification to the monitoring team.
2) Error audit - verifies whether processes around the system tasks that run on timer are executed correctly. Once the issue has been identified, the system sends an email notification to the monitoring team.

Intended Audience

This guide will be useful for administrative users and system administrators that receive monitoring system audit and/or system administration email notifications and gives knowledge how to manage occurred system issues at a higher level.

System Audit Notifications

The system audit processes are running based on three cycles:

• Daily – executes once a day;
• Hourly – executes every hour;
• Quarter-hourly – executes every 15 minutes.

The following audit procedures are executed as part of these cycles. The current list of the audit procedures is compiled based on the operational experience with the gateway and designed to minimize disruption from potential bugs or user errors within the critical parts of the system. First digit of the procedure ID indicates the type of the problem occurred:
1xx - merchant/reseller/portfolio management issues;
2xx - processing issues;
3xx - system issues.

To learn how to set the recipients of the system audit notifications, review this tutorial. The following options are available for setting recipients of the notifications:

Daily Audit Procedures

1. Merchants with duplicate identifiers (ID: 101)

Definition:
There is a system capability allowing users to assign custom identifiers for various merchant records. There are some situations when the merchants with duplicate identifiers might be present in the system. The purpose of this procedure is to identify any potential duplicates for subsequent resolution.
Action:
Change a merchant code value for a merchant that has a duplicate:
1. Go to the Merchant perspective.
2. Select a merchant.
3. Click on Details button in the top section of the screen.
4. Select General menu item, as on the screenshot.
5. Change value in Code textbox.

2. Accounts with duplicate identifiers (ID: 102)

Definition:
There is a system capability allowing users to assign custom identifiers for various account records. There are some situations when the accounts with duplicate identifiers might be present in the system. The purpose of this procedure is to identify any potential duplicates for subsequent resolution.
Action:
1. Go to the Merchant perspective.
2. Select a merchant and an account, associated with this merchant.
3. Click on Details button on the top section of the screen.
4. Select General menu item, as on the screenshot.
5. Change value in Code textbox.

3. Users pending approval (ID:103)

Definition:
When new users are created using the Console perspective based on the ZK 8.6.3 version, they obtain the "Pending Approval" status and are not active yet within the gateway. The purpose of this procedure is to notify system administrators, whether any pending users are present. These notifications include a new user's username, full name and name of the user's owner (for example, a merchant that had created the user).
Action:
1. Log in to the gateway and check if all information in the user's settings is correct.
2. Further action depends on the results of the user's information verification:
a) If the user's information is valid, set user's status as Active.
b) If the user's information is invalid, set user's status as Unapproved.

4. Non-tokenized card numbers (ID: 201)

Definition:
There might be situations, such as emergency processing or user mistakes in configurations, when non-tokenized (encrypted) card numbers might get stored in the database. This procedure allows to identify such transactions and notify administrative users of their presence. Under normal circumstances, daily tokenization process converts such card numbers into tokens if they were not converting originally.
Action:
1. Check URL where all the transactions are submitted to.
2. If the URL is wrong, change it for an appropriate one.

5. Blacklist integrity (ID: 202)

Definition:
The system maintains an internal blacklist for credit cards hard declines and direct debit returns. There might be situations when account numbers, for which such returns or declines occurred, were not properly added to the blacklist. This procedure is designed to identify two things: 1) accounts that were supposed to be added to the blacklist, but did not get added properly; 2) transactions, that were supposed to be blacklisted, but were actually submitted for processing.
Action:
1. Check whether a credit card or direct debit account is added to the blacklist (to learn how to check blacklisted accounts, review this tutorial).
2. If the credit card or direct debit account is not in the blacklist, check the statuses of the transactions from the system audit message (to learn how to locate a transaction, review this tutorial).
3. In case you cannot resolve the issue on your own, contact gateway support.

6. Decline rate deviations of real-time transactions (ID: 203)

Definition:
Under normal circumstances, the decline rate for real-time transactions for any given merchant with substantial volume (over 10 transactions) should not exceed 15%. The purpose of this procedure is to identify merchants and accounts that have an unusually high decline rate of real-time transactions.
Action:
1. Locate declined transactions associated with a merchant present in the system audit message (to learn how to locate transaction, review this tutorial).
2. Check the response codes and response messages associated with these transactions.
3. If the code and message indicate that there are issues on the processor's side, address the issue to the processor.
4. In case you cannot resolve the issue on your own, contact gateway support.

7. Files (batch) validation errors (ID: 207)

Definition:
Within the context of batch processing, there are situations when transactions, included in a batch file, have critical errors and validation file is generated by the system. The purpose of this procedure is to identify all validation files generated by the system and provide this information to the system administrators and monitoring team.
Action:
1. Check indicated validation files on FTP server and the user interface (to learn how to locate validation files, review this tutorial).
2. Add corrections for files if needed.
3. Resubmit these files via either the user interface or FTP server.

8. Chargeback integrity (ID: 209)

Definition:
There are cases when chargeback information from the processor is incomplete, and missing information might affect the users visibility and understanding of the situation related to chargebacks. This procedure is designed to identify potential incomplete cases, where some critical information from the processor was not provided.
Action:
1. Check the status of the chargeback case from the system audit message (to learn how to locate a chargeback, review this tutorial).
2. Review whether all information regarding the chargeback case is provided.
3. In case you cannot resolve the issue on your own, contact gateway support.

9. Chargeback document integrity (ID: 210)

Definition:
There are situations when during the upload of supporting documentation for a chargeback, an error occurs due to file size or a total number of files, or other conditions. This procedure is designed to identify any failed uploads that may have potentially prevented the representment process from completing and allows to assist a merchant with the upload process and the representment of the chargeback in a timely fashion.
Action:
1. Locate the chargeback case from the system audit message (to learn how to locate a chargeback, review this tutorial).
2. Check whether all necessary documentation is uploaded.
3. In case you cannot resolve the issue on your own, contact gateway support.

10. Settlement reject verification (ID: 211)

Definition:
When Terminal Capture mode is used for settlement, there might be situations when certain transactions are rejected at the settlement time. Such transactions are recorded in the system as a separate transaction type 'Reject'. The purpose of this procedure is to identify such rejected transactions for further analysis of the rejection cause.
Action:
1. Review the return file (to learn how to locate a return file, review this tutorial).
2. Export the information about rejected transactions from the return file and address it to the processor.

11. Missed settlement verification (ID: 212)

Definition:
There is a system capability to settle batch manually through the UI or API. Consequently, when it is enabled, there might be situations when certain batches are not settled and authorizations might expire. The purpose of this procedure is to identify such unsettled batches to prevent associated authorizations from expiring. These notifications include merchant's credentials (merchant name and account ID) and information about the cycle that experienced the issue (cycle ID, open date and close date).
Action:
1. Identify an unsettled batch within the system, using ID from the system audit message.
2. Initiate authorized amount is withdrawn from the card holder’s account and transferred to merchant’s account. The general practice is to do this at the end of the business day. There are two possible settlement mechanisms commonly referred to as terminal capture and host capture: When terminal capture is used, the information about each transaction to include in settlement has to be supplied at the settlement time (generally through a settlement file). When host capture is used, the underlying processing system (the host) keeps track of all of the transactions and it is usually sufficient to simply send a settlement message without including transaction details. " >settlement for this batch (to learn how to initiate settlement for batches, review this tutorial).
Note: if a merchant is configured with manual realtime settlement, a realtime batch should be approved within two days after submission, using the user interface or via API.

12. Chargeback case integrity (ID: 213)

Definition:
Within the context of chargeback processing, there are situations when some chargeback case data can be missing in the database after the case is submitted. The purpose of this procedure is to verify data integrity and provide information about missing information to system administrators.
Action:
In case you cannot resolve the issue on your own, contact gateway support.

13. Failed chargeback case document upload (ID: 214)

Definition:
Within the context of chargeback processing, there are situations when some issues occur while the uploading of chargeback related documents to a processor’s server. The purpose of this procedure is to identify the documents that did not get uploaded and provide this information to system administrators.
Action:
1. Check if the format, size and other parameters of the document meet the requirements of the processor.
2. Check if the card type associated with a chargeback case is supported for processing by the processor.
3. In case you cannot resolve the issue on your own, contact gateway support.

14. Expiring private keys (ID: 215)

Definition:
When private keys for data encryption and decryption are generated, expiration period is set for them. In cases when a specific private key is about to expire, the system will notify system administrators of the upcoming expiration. The purpose of this procedure is to identify keys that are about to expire and notify administrators in advance to give them time to replace keys accordingly.
Action:
In case you cannot resolve the issue on your own, contact gateway support.

15. Expiring certificates (ID: 216)

Definition:
Within the context of the secure HTTP communication (real-time processing), the system must have a specific certificate installed to be able to connect to remote server. In cases when a certificate is about to expire, the system will notify system administrators about the upcoming expiration. The purpose of this procedure is to identify certificates that are about to expire and notify administrators in advance to give them time to replace them accordingly.
Action:
In case you cannot resolve the issue on your own, contact gateway support.

16. CAU transaction discrepancies (ID: 217)

Definition:
Within the context of Credit Card Account Updater, there are situations when updates identified by response code do not correspond to the updated values provided (for example, response suggests that card number was changed, but expiration date updated values do not correspond). The purpose of this procedure is to identify such situations and notify administrative users about them for subsequent review.
Action:
In case you cannot resolve the issue on your own, contact gateway support.

17. Failed convenience fee processing (ID: 218)

Definition:
Within the context of convenience fee processing, there are situations when transactions with convenience fee get failed. The purpose of this procedure is to identify these transactions and provide this information to the system administrators.
Action:
In case you cannot resolve the issue on your own, contact gateway support.

18. Decline rate deviations of batch transactions (ID: 221)

Definition:
Under normal circumstances, the decline rate for batch transactions for any given merchant with substantial volume (over 10 transactions) should not exceed 60%. The purpose of this procedure is to identify merchants and accounts that have an unusually high decline rate of batch transactions. These notifications include merchant credentials (name and account ID), name of the reseller (if present), name of the provider profile, code of the batch, number of transactions/approvals/declines and decline rate.
Action:
1. Locate declined transactions associated with a merchant present in the system audit message (to learn how to locate transaction, review this tutorial).
2. Check the response codes and response messages associated with these transactions.
3. If the code and message indicate that there are issues on the processor's side, address the issue to the processor.
4. In case you cannot resolve the issue on your own, contact gateway support.

19. Expired terminal P2PE keys (ID: 222)

Definition:
Within the context of terminal transaction processing, P2PE keys should be rotated every 18-24 months. The purpose of this procedure is to identify terminals which did not have their P2PE keys updated for more than 7 days after the due period and notify administrative users about those terminals. These notifications include terminal credentials (terminal ID, account ID, merchant name) and a date when P2PE RSA keys rotation request was received (at least 7 days prior to the current date).
Action:
1. Check if the terminal was active during the key requesting process as keys can be rotated to active terminals only.
2. If a terminal was inactive, re-submit the API call for a P2PE key request.
2. In case you cannot resolve the issue on your own, contact gateway support.

20. Unscheduled terminal restart (ID: 224)

Definition:
Within the context of terminal transaction processing, there are situations when the unscheduled reboot of a terminal occurs. The purpose of this procedure is to identify terminals that experienced unplanned restarts within a business day and notify administrative users about them. These notifications include terminal credentials (terminal ID, account ID, merchant name) and a total number of occurred restarts.
Action:
1. Check the log files associated with this terminal for any malfunction indicators.
2. In case you cannot resolve the issue on your own, contact gateway support.

21. Duplicate transactions (ID: 225)

Definition:
Within the context of the batch processing, there are situations when a processor marks certain transactions as duplicates. The purpose of this procedure is to identify such reported duplicates and notify administrative users for subsequent review.
Action:
1. Check whether a specified batch is duplicate or not (to learn how to locate a duplicate batch, review this tutorial).
2. If the batch is a duplicate, cancel it.
3. If it is not a duplicate, it should be approved for further processing.

22. Unmapped transaction declines (ID: 226)

Definition:
Within the context of transaction processing, there are cases when processors return response codes that were not previously used or were not listed in the original integration specification. Such codes are mapped into generic declines - such as D19 for credit card transactions and R97 for direct debit transactions. The purpose of this procedure is to identify such declines so the proper mapping logic for the new codes can be put in place. These notifications include response code, number of declines, accociated account ID and transaction ID.
Action:
1. Address the issue to the processor to check what response code corresponds to the one that is present in the system audit message.
2. Address the information received from the processor to the gateway support.

23. Unmapped token declines (ID: 227)

Definition:
Within the context of operations with tokens (tokenization/detokenization/untokenization/retokenization), there are cases when processors return response codes that were not previously used or were not listed in the original integration specification. Such codes are mapped into generic P19 decline. The purpose of this procedure is to identify such declines so the proper mapping logic for the new codes can be put in place.
Action:
1. Address the issue to the processor to check what response code corresponds to the one that is present in the system audit message.
2. Address the information received from the processor to the gateway support.

24. Failed reversal transactions (ID: 228)

Definition:
Within the context of sale-auth transaction processing, there are situations when a processor rejects voided transactions. The purpose of this procedure is to identify all failed reversals and provide this information to the system administrators and monitoring team. These notifications include merchant credentials (account ID, merchant name, and billing profile type), reseller's name and transaction ID.
Action:
1. Locate the void transactions from notification using their IDs.
2. Issue a manual refund for each transaction (to learn how to refund transactions, review this tutorial).

25. Tokenization profile synchronization errors (ID: 229)

Definition:
Within the context of customer profiling related to tokenization process, there are situations when a processor updates a customer’s profile associated with a particular token within the system, and this token profile does not get synchronized with the gateway. The purpose of this procedure is to identify tokens that failed to update and provide this information to the system administrators and monitoring team.
Action:
1. Identify the issue using a response code and response message received in the notification.
2. If the issue is related to the connection problems or an invalid value of a particular field, update a token’s profile via submitting retokenization API call.
3. In case you cannot resolve the issue on your own, contact gateway support.

26. Suspended terminal update (ID: 231)

Definition:
Within the context of the terminal update process, there are situations when a terminal update cannot be completed. When a terminal fails to be updated for five times, an update mode of the terminal changes to Suspended. The purpose of this procedure is to identify all terminals with suspended update mode and provide this information to the system administrators and monitoring team. These notifications include merchant credentials (account ID and merchant name) and terminal ID.
Action:
1. In the terminal details on the user interface, change Update Mode to Enabled and save the changes.
2. Check if there is an issue with TMS server connection.
3. Download the terminal log file to review it for any issues that could cause the problem (to learn how to locate terminal logs, review this tutorial).
4. In case you cannot resolve the issue on your own, contact gateway support.

27. Unprocessed onboarding applications (ID: 232)

Definition:
Within the context of merchant onboarding, there are cases when a processor that performs onboarding does not approve the onboarding application during a business day.
While Camel process is running the onboarding application response code can take one of the following values (Decline, Review, Needs Correction, Approved Warning). To make the Daily System Audit section more informative in Merchant Onboarding report the onboarding applications are grouped by providers where their notifications are divided by onboarding application statuses from highest to lowest: 1) Needs Correction; 2) Approved Warning; 3) Error; 4) Declined; 5) Failed; 6) In Review. The purpose of this procedure is to identify applications that are in one of the following statuses (Decline, Review, Needs Correction, Approved Warning), and provide this information to the monitoring team. In addition, it provides reporting on onboarding applications grouped by the processor and the application status in the descending order (from the status with the highest priority to the lowest).

Action:
1. Check the status of the listed application(s) on the user interface.
2. If the application has one of these statuses (Needs Correction, Approved Warning, Decline, In Review) follow these steps :

• Refresh its status by clicking on the Status Update button. If the status has not changed, continue to step 3.
• If the Background Verification option is available for the processor, review the details of the processor's response via a corresponding form. Once the issue is corrected, re-submit the application.

3. If no additional details regarding the reason why the application cannot be approved are not provided, address the issue to the processor.
4. If the status of the application differs from the one on the processor's side, contact the gateway support.

28. Missing processing dates (ID:233)

Definition:
Within the context of realtime and batch processing, there are situations when the processing dates are missing in separate transactions or submissions in batch file. The purpose of this procedure is to identify such records in the system for further analysis and fixing the possible issues.
Action:
In case you cannot resolve the issue on your own, contact gateway support.

29. Server information (ID: 301)

Definition:
Under normal circumstances, there is a set of characteristics that indicate a current state of a server where the application operates on. The purpose of this procedure is to summarize a current state of these characteristics and provide this information to the system administrators and monitoring team. These notifications include an application version, cluster status, the number and name of a node, as well as an indicator of whether debug logging is enabled.
Action:
In case you cannot resolve the issue on your own, contact gateway support.

30. Unassigned Terminal Configurations (ID: 234)

Definition:
Every time when new CA keys for a particular processor are uploaded to the system, an updated Contact/Contactless EMV configuration record is created.
This record contains the configuration file with the up-to-date CA keys. After this, the reference to the configuration record in Provider database table must be updated.
The process of configuration record update (CAKeysChange) implies the creation and activation of a new record with the relevant configuration file, deactivation of all other records of a particular type (Contact/Contactless) for a given processor. The purpose of this notification is to notify the system administrators and monitoring team about the new terminal configuration. These notifications include Provider Code, Provider Name, Configuration Type (Contact/Contactless), Create Date, Description (tag).
Action:
1. Navigate to System Perspective/Providers, locate the processor for which the CA keys have been updated and click magnifying glass to open Provider Modify form.
2. Check the last EMV Contact/Contactless configuration record:

• make sure that the Active field value is set to Yes;
• make sure that the description field value begins with #CAKeysChange.
• verify the information about the configuration file on the corresponding View form: make sure that none of the fields are empty or equal to zero;
• activate the record in EMV Configuration subsection of the Provider Modify form (in the menu you will see only active Contact/Contactless configurations);
• save the changes;
• if the changes have been saved successfully, clear the Description field for this record and save the changes in the record.

Hourly Audit Procedures

1. Processing interruptions (ID: 223)

Definition:
Within the context of velocity tracking mechanism, there are situations, when due to various network conditions or errors, consistent stream of transaction processing through a particular channel, merchant or reseller is interrupted. In such case, to track these situations, it is possible to set minimum hourly transactions limit for the channel, or merchant, or reseller. The purpose of this procedure is to notify system administrators, if the minimum transaction count is not reached during the hour, for subsequent review.
Action:
1. Check if there is an issue with the connection.
2. If the connection is stable, address the issue to the processor.
3. In case you cannot resolve the issue on your own, contact gateway support.

Quarter-hourly Audit Procedures

1. Files (batch) processing errors (ID: 204)

Definition:
Under normal circumstances, a file is processed within at most four hours after submission. Processed status indicates that processing of the file is complete. In cases when a file status was not changed to Processed after four hours, the system sends the notification to system administrators. If the file wasn't processed, its possible statuses can be Download (when the file is waiting for processing) and Backward (when the file has problems with data and failed on a certain phase of processing).
Action:
Request processor for further clarification on the issue.

2. Configuration communication declines (ID: 205)

Definition:
There are decline codes that come from the processor to indicate incorrect merchant ID or terminal ID configuration or some communication problems. The purpose of this procedure is to identify such cases and take immediate action to restore processing.
Action:
1. Review merchant's configuration to identify if there is an issue. The configuration has to be reviewed using Merchant perspective => Details => Profile => {profile that the issue was reported for}.
2. If a configuration is correct, address the issue to the processor.
3. In case you cannot resolve the issue on your own, contact gateway support.

3. Files (batch) validation errors (ID: 208)

Definition:
Within the context of batch processing, there are situations when transactions, included in a batch file, have critical errors and validation file is generated by the system. The purpose of this procedure is to identify all validation files generated by the system and provide this information to system administrators and monitoring team.
Action:
1. Check indicated validation files on FTP server and the user interface (to learn how to locate validation files, review this tutorial).
2. Add corrections for files if needed.
3. Resubmit these files via either user interface or FTP server.

4. Unarchived job records (ID: 230)

Definition:
Under normal circumstances, the execution of all jobs that are run by timer is archived and stored in the database. In case if any executed or canceled tasks have not been recorded, the system sends the notification to the system administrators. The purpose of this procedure is to verify whether tasks archiving is active within the system.

Action:
1. Check whether a corresponding timer used for tasks archiving is active.
2. If the timer is inactive, activate it and restart Camel.
3. In case you cannot resolve the issue on your own, contact gateway support.

5. System error log analysis (ID: 302)

Definition:
Within the context of real-time processing, there are situations when submitters of transactions experience various types of issues due to either an error in the request, incorrect configuration of an account within the gateway, or other types of the gateway-related problems. All such errors are logged into a designated internal log table. The purpose of this procedure is to identify the records in this log and, if any records are present, notify system administrators about them. These notifications include the submitter's credentials (username and account ID), terminal ID (if involved in the issue), code of the node, error code and error message. All records are grouped into sections according to their type, error code and message. If the error's code has not been identified, it is placed in the Other section.
Action:
1. Review the error code and message included in the notification.
2. In case if you see an unfamiliar error code/message, go to the API documentation and review the error codes section. We recommend to pay close attention to the S25 ProcessName process failed. Contact support. error as it might indicate that some of the critical processes stopped working and requires immediate resolution.
3. To find information about the code, use the searchbox.
4. In case you have not found the code, contact gateway support.

6. Disc space errors (ID: 303)

Definition:
The system is able to check the availability of hard drive space that is required for normal system operation of TidyPay. When a hard drive is running out of space, the system sends emergency notifications to system administrators.
Action:
1. Check the place available on the hard drive.
2. Clean the space if it is possible.
3. If the hard drive cannot be cleaned, make sure that you have an extra hard drive available.

7. Log size increase errors (ID: 304)

Definition:
Under normal circumstances, the size of a log file does not increase immensely within a short period of time. The purpose of this procedure is to notify system administrators when the size of a log file was increased on more than 100 Mb within 15 minutes (after the previous system audit was made) because such issue can cause the lack of the free space on disc with TidyPay installed.
Action:
1. Download the log file that has been increasing too fast (to learn how to find logs, review this tutorial).
2. Check the log file to investigate the origin of the issue.
3. In case you cannot resolve the issue on your own, contact gateway support.

8. System warning log analysis (ID: 305)

Definition:
Within the context of real-time processing, there are situations when submitters of transactions experience various types of not critical, but important issues. For example, if a terminal failed to process transactions due to incorrect terminal parameters, or if transaction processing took more time than expected. All such errors are logged into a designated internal log table. The purpose of this procedure is to identify the records in this log and, if any records are present, notify system administrators about them.
These notifications include the submitter's credentials (username and account ID), terminal ID (if involved in the issue), code of the node, error code (always starts with “W”) and error message. Records are grouped if they have similar values for all columns.
Action:
1. Review the error code and message included in the notification.
2. In case if you see an unfamiliar error code/message, go to the API documentation and review the system codes section.
3. To find information about the code, use the searchbox.
4. In case you have not found the code, contact gateway support.

9. Camel Error Log (ID: 306)

Definition:
Within the context of batch processing, there are cases when Camel jobs fail and corresponding UniPay exceptions are generated. The purpose of this procedure is to notify system administrators about failed Camel batch jobs and to provide them with the error details necessary for further root cause analysis. These notifications include failed job name, information on the related job objects (IDs, names), node on which the error ocurred, error code and error message.

Action:
1. Review the error code and message included in the notification.
2. Analyze and eliminate the root cause of the issue.
3. Requeue the failed job if necessary.
4. In case you could not fix the issue, contact gateway support.

Error Audit Notifications

Transaction processing issues that are related to the system monitoring can be divided in three groups – file delivery via FTP server, database related issues and configuration errors.

• FTP server issues - are usually connected to the issues on the processor’s side (processing within file store) and issues related to the closing of retail transaction cycle.
• Database issues - are usually related to the connectivity between the database and the gateway.
• Configuration errors - are usually related to the incorrect configuration settings applied within the system.

Main identifiers of the described issues and the ways how to investigate and resolve them are listed below.

File delivery issues via FTP server

1. File uploading

Definition:
There can be cases when it is not possible to upload a file to the FTP server of a processor. In this case, a respective email from the system is received. The purpose of this procedure is to provide the information about the place, time, object and merchant ID that experiences the issue. It also includes the trace information that allows to identify what processes have failed.

Symptoms:
Let’s review an example of an email reporting about the issue.

This part of the email allows us to get general information about the issue:

The message includes the details that allow us to identify the type of the issue that failed. The highlighted values are the most important while analyzing the issue:

Message:
The final part of the email includes stack trace information about the actual components that are involved in the issue. Stack trace analysis allows to have deeper understanding of the issue.


Action:
The reasons and the ways of resolution of an issue can be the following:

1.1. Configuration of profile or FTP server settings

Processor profile or FTP server settings are not configured properly – invalid information is included.

You can identify this problem by stack trace content:
To resolve this issue, review if all data in the processor’s profile and FTP server data is relevant.

1.2. Access blocked by firewall

Processor has no access to the server because of the firewall settings that blocked the external access.

You can identify this problem by stack trace content:
To resolve this issue, check the firewall settings and exclude the processor’s IP from the firewall blacklist.

1.3. Access blocked due to incorrect IP

There is no access to the processor’s server, because gateway server’s IP was previously changed and the processor was not informed about that.

You can identify this problem by stack trace content:
To resolve this issue, reach out the processor, requesting a new IP to be added to the whitelist.

1.4. Locked account due to duplicate suspicion

FTP server account is locked by a processor. This situation can occur in cases when previously submitted and declined transactions are sent several times (merchant doesn’t have retry mechanism enabled). Processors can block an FTP server account because of the duplicate suspicion.

You can identify this problem by stack trace content:
To resolve this issue, contact a processor.

1.5. Denial for file storage on the server

You don't have permission on file storage on the processor's server. This problem can occur when the inbound and outbound folders are not set or are set improperly.

You can identify this problem by stack trace content:
To resolve this issue, make sure that the respective processor profile settings are correct.

1.6. Unsuccessful processing of transaction with invalid routing number

A direct debit transaction cannot be processed because of the invalid format of accountAccessory (routing number).

You can identify this problem by stack trace content:
To resolve this issue, address a request to the merchant to reupload the transaction using a correct accountAccessory format (to learn more about the formats of accountAccessory, review this note).


Note: After the problem with file uploading is resolved, the file must be reprocessed (review these tutorials to learn how to restart the processing of a file).

2. File downloading

Definition:
There can be cases when it is not possible to download a response file from a processor’s FTP server. In this case, a respective email from the system is received. The purpose of this procedure is to provide the information about the place, time, object and merchant ID that experiences the issue. It also includes the trace information that allows to identify what processes have failed.

Symptoms:
Let’s review an example of an email reporting about the issue.

This part of the email allows us to get general information about the issue:
The message includes the details that allow us to identify the type of the issue that failed. The highlighted values are the most important while analyzing the issue:

Message:
The final part of the email includes stack trace information about the actual components that are involved in the issue. Stack trace analysis allows to have deeper understanding of the issue.

Action:
The reasons and the ways of resolution of an issue can be the following:

2.1. Access denied due to overload of the processor

Processor is overloaded with requests and, as a result, access was denied.

You can identify this problem by stack trace content:
To resolve this issue, restart the failed task after some period of time. To learn how to restart the processing of a file, review these tutorials.

2.2. Processing failed due to overload of files

The process failed due to big amount of files being downloaded.

You can identify this problem by stack trace content:
To resolve this issue, restart the failed task after some period of time. To learn how to restart the tasks, review these tutorials.

2.3. FTP server locked by a processor

FTP server account is locked by a processor for some reason.

You can identify this problem by stack trace content:
To resolve this issue, contact a processor. When the access is allowed, reprocess a request. To learn how to restart the processing of a file, review these tutorials.

3. Settlement issues

Definition:
There can be cases when a retail transaction cycle is not closed successfully. In this case, a respective email from the system is received. The purpose of this procedure is to provide the information about the place, time, object and merchant ID that experiences the issue. It also includes the trace information that allows to identify what processes have failed.

Symptoms:
Let’s review an example of an email reporting about the issue.

This part of the email allows us to get general information about the issue:
The message includes the details that allow us to identify the type of the issue that failed. The highlighted values are the most important while analyzing the issue:

Message:
The final part of the email includes stack trace information about the actual components that are involved in the issue. Stack trace analysis allows to have deeper understanding of the issue.

Action:
The reasons and the ways of resolution of an issue can be the following:

3.1. Inactive account

An account has been deactivated before its transaction cycle was closed in the gateway.

You can identify this problem by stack trace content:
To resolve this issue, activate a respective account (review this tutorial to learn how to activate/deactivate an account) and close an RTC via API (review this information to learn how to use the API commands; close-cycle operation should be used)

3.2. Void issues

1) A voided transaction is not processed successfully.

You can identify this problem by stack trace content:
To resolve this issue, set the cut-off time for a respective merchant at least 15 minutes earlier than a processor's cut-off time and make a refund for transaction with the issue.

2) Gateway cannot cancel the unconfirmed transaction automatically.

You can identify this problem by stack trace content:
To resolve this issue, check whether all required fields are present in a processor's profile. After the profile is verified and updated (if needed), restart the failed task or it will be restarted by the system automatically.

3) An unconfirmed transaction cannot be voided because a processor has already settled this transaction.

You can identify this problem by stack trace content:
To resolve this issue, issue refund or create credit transaction with same amount to return money to the customer.

3.3. Failed sale-auth transactions due to the profile change

Sale-auth transactions got failed to process due to changes in a processor's profile.

You can identify this problem by stack trace content:
To resolve this issue, revert previous settings back and confirm sale-auth transactions.

Database connection issues

Definition:
There can be cases when the connection to a server, where database is located, is failed. In this case, a respective email from the system is received. The purpose of this procedure is to provide the information about the place, time, object and merchant ID that experiences the issue. It also includes the trace information that allows to identify what processes have failed.

Symptoms:
Let’s review an example of an email reporting about the issue.

This part of the email allows us to get general information about the issue:
The message includes the details that allow us to identify the type of the issue that failed. The highlighted values are the most important while analyzing the issue:

Message:
The final part of the email includes stack trace information about the actual components that are involved in the issue. Stack trace analysis allows to have deeper understanding of the issue.

Action:
The reasons and the ways of resolution of an issue can be the following:

1. System changes while being operating

Some part of the system that is connected to the database, got changed or updated, while gateway was operating. When any changes are applied to the database associated software/hardware or the server, the gateway operation should be put on hold.

You can identify this problem by stack trace content:
To resolve this issue, restart the task after all the changes are completed to the system.

2. Failed DNS connection

DNS connection between the database and gateway servers got failed.

You can identify this problem by stack trace content:
To resolve this issue, check the connection settings.

3. No free space on the hard drive

The problems are within the database. For example, hard drive is running out of space.

You can identify this problem by stack trace content:
To resolve this issue, check the place available on the hard drive.

Configuration issues

Definition:
There can be cases when merchant configuration does not allow to fulfil a particular operation within the gateway. In this case, a respective email from the system is received. The purpose of this procedure is to provide the information about the place, time, object and merchant ID that experiences the issue. It also includes the trace information that allows to identify what processes have failed.

Symptoms:
Let’s review an example of an email reporting about the issue.

This part of the email allows us to get general information about the issue:
The message includes the details that allow us to identify the type of the issue that failed. The highlighted values are the most important while analyzing the issue:

Message:
The final part of the email includes stack trace information about the actual components that are involved in the issue. Stack trace analysis allows to have deeper understanding of the issue.

Action:
The reasons and the ways of resolution of an issue can be the following:

1. Generation of merchant statements for inactive merchant

Merchant statement is generated while a respective merchant is inactive.

You can identify this problem by stack trace content:
To resolve this issue, check merchant configuration (review this tutorial to learn how to review merchant configuration). If the merchant was deactivated mistakenly, activate this merchant within the gateway.