Genesys: Solving Config Server in read-only mode

Product: Genesys CTI Suite 7.x to 8
Module: Configuration Server

Under unknown scenario, Genesys Config server may starts and complain another instance of Config server is running. This is more often in HA environment where there are 2 machine running primary/backup Config server.

This post assume you have done all checks and verification for backup Config server, and verified Task Manager that no other Config server is running, and certainly not network or database issue.

Common message in Config server's log: Another Configuration Server is detected

Option 1: Possibility of Windows run away process
1. Try reboot Windows if possible. If can't, skip this. For Windows, you can't trust Task Manager, but use Process Explorer to verify whether Config server is really terminated

Option 2: Possibility of backup Config server improperly switching to primary role, but in transient and never complete the process
2. Shutdown backup Config server, then restart primary Config server

Option 3: DB Server used by Config server is passing wrong info
1. Restart Genesys DB Server used by Config server. Verified Connectivity tab of Config server to determine the name of the DB Server
2. Then restart primary Config server

Option 4: Possibility Oracle listener down
1. Checks to ensure Oracle listener is up (if using Oracle database)
2. Oracle database and listener are 2 different process. So database up does not mean listener is up, or able to connect (if listener setup for RAC, fail-over, load balancing)
3. Runs SQL*Plus from Genesys DB Server (the one used by Config server) and login with the username and password configured in CME to ensure it is able to connect. Assume Oracle is not install in the same box as DB Server
4. If DB Server and Oracle database server install in same box, then use following command to test

sqlplus username/password@configdbprod

Note: Fill in username and password with actual username. Replace database listener name configdbprod with actual listener name

5. Listener name above is may not be the database name, although it is often configured so. If using RAC, then there are many more names. Used the name that configured to used by DB Server. If have problem then test other name to determine the root cause. Ask me if you need assistant in Genesys with Oracle RAC database environment

Option 4: Table config.cfg_refresh contains invalid value that confused Config server
1. Verify the content of table cfg_refresh. Ask me if you need to know how to find the database and query it. I am expert in database as well
2. During every shutdown of primary and backup Config server, verify its content before and after
3. During every startup of primary and backup Config server, verify its content
4. While keeping Config server down, note down its value (it is only 2 columns for ver 7.x) before making any change, but after you notified its pattern
5. Don't modify value for column NOTIFY_ID
6. Change value of column REFERSH

update cfg_refresh set refresh = 0

7. Startup primary Config server

Long term solution:
1. Upgrade to Config Server 8.0.300.30 or higher. This is a product bug

Please use following PayPal donate if my post helped

Genesys ETL Run-time contains inappropriate object or statistic

Product: Genesys CTI
Module: ETL Runtime Service

Encounter following error in log ETL_Service.20121213_214131_050.log?

15:39:30.593 Src.ChunkData.1 Chunk#1_2075 contains inappropriate object or statistic


Following is the step to troubleshoot

Connects to ODS database

Find out the schedule # from chunk #
select  * from ol_chunk_log where log_id = 2075


LOG_ID SCHEDULE_ID BEGIN_TIME              UPDATE_TIME             CHUNK_TYPE TABLE_NAME IS_LAST_CHUNK RECEIVE_TIME            TRANSFERRED_TIME        DELETE_TIME IS_NULL_CHUNK DATA_RECORDS_NUM
------ ----------- ----------------------- ----------------------- ---------- ---------- ------------- ----------------------- ----------------------- ----------- ------------- ----------------
2075   9           2010-09-26 20:00:00.000 2010-09-26 20:15:00.000 NULL       OL_DATA3   0             2010-09-26 20:15:02.000 2010-09-26 16:19:06.560 NULL        0             227

Its value is 9

Use schedule # to find layout #
select * from ol_schedule where schedule_id = 9


SCHEDULE_ID LAYOUT_ID TIME_PROFILE_ID START_TIME              STOP_TIME REFRESH_RATE IS_NEEDS_TRANSFER FREEZE_TIME
----------- --------- --------------- ----------------------- --------- ------------ ----------------- -----------------------
9           9         1               2010-06-07 13:36:26.670 NULL      NULL         1                 2010-06-07 13:48:07.297

Its value is 9. This value is very often same as schedule #

select * from ol_report_layout where layout_id = 9


LAYOUT_ID OBJECT_TYPE_ID TENANT_ID LAYOUT_NAME    METAGROUP_CLASS METAGROUP_DB_ID ADD_TIME                DELETE_TIME             LAYOUT_DESCRIPTION           UPDATE_TIME TEMPLATE_NAME IS_BRKN
--------- -------------- --------- -------------- --------------- --------------- ----------------------- ----------------------- ---------------------------- ----------- ------------- -------
9         0              101       Agent_Layout_2 100             0               2010-06-10 11:42:40.147 2010-10-13 16:59:08.983 Agent Report Layout with AHT NULL        AGENT_AHT1    NULL

This means the chunk belongs to Report Layout Agent_Layout_2 (see DMA), and time of the record is 2010-06-07 13:36:36

Normally this error appear if report layout is deleted from DMA before ETL service able to pull the data into DATAMART/REPORT database. If you confirmed these can be deleted, then take a backup of the ODS database and execute following statement to delete all the entries belongs to "Agent_Layout_2"

detete from ol_report_layout where layout_id = 0

This is no way to undo from DMA, or ETL service. The only way to undo is to restore from database backup, or flashback (Oracle) or similar technology offer by database server

Genesys Outbound OCM: Control Campaign Viewing Privilege/Permission

Product: Genesys CTI Suite 5.x - 8.x
Module: Outbound Contact Manager (OCM)

In order to limit different agents to see their own campaigns, campaign groups and calling lists, following are require (use CME)

1. Create an Access Group, says HR_AccGrp. Grant this Access Group to the selected agent
2. Create a sub-folder under Table Access to keep the selective table access objects together for ease of management, and privilege management
3. Create a sub-folder for Calling List like above
4. Create a sub-folder for Campaigns like above
5. Change privilege of root folder Table Access, Calling List, Campaigns, Agent Groups (if any) to have read + execute access by HR_AccGrp. Do not propagate the change to sub-folder else they will have full visibility, which is not desire
6. Change privilege of each of the sub-folder created for Table Access, Calling List, Campaigns, Agent Groups (if any) to have read+execute privilege access by HR_AccGrp
7. Launch OCM with the agent login to verify the account can only see the campaigns, calling lists, and agent groups defined in the sub-folder

If any privilege is missing, following error will appear in OCM after it prompts for OCS server

Error reading configuration

Other related objects that may need to verify, if someone purposely revoke the privileges are


Treatment
Filter
Time Zones
Fields
Format

Remember that both root folder, sub-folders and the objects itself contains the privilege. Think about the privilege in both CME and OCM as outbound manager may need access to CME to administrate their own campaigns as well, like user management among several campaigns

Crystal Report: Reasons for "Encapsulation Page Error"

Product: BusinessObject Crystal Report

Following are possible cause for "Encapsulating Page Failed"

1. If uses sub-report, ensure sub-report's database authentication is correct
2. If rpt file is from other database environment, or not from original designer, open the rpt with Crystal Report designer, then execute it to identify any database related error. Possible errors
2.1. View does not exists
2.2. Column does not exists
2.3. Database function does not exists
2.4. No privilege to access table or view which get false alarm that view or table does not exists
2.5. Database account locked, or password expired
2.6. Other database related error
3. If using NTFS, ensure BusinessObject installation directory has proper privilege. Re-install the product if can't identify the exact NTFS privilege
4. Bug in Crystal Report server. Install the latest Service Pack to the server, not to Crystal Report designer

Please use following PayPal donate if my post helped

Genesys - Server with this name is already running

Product: Genesys CTI Suite
Module: Genesys CPD 8.0.001.02

There are various ways that CPD server may lost connection to Genesys config server. While many sites covering normal scenarios, such as config server down, database connectivity issue which cause config server failed to start, CPU bottleneck in CPD or config server, RAM bottleneck in CPD or config server, CPD 32-bit version hits 2 GB/process max, network instability, etc. This post is covering an unknown scenario.

For unknown reason, when a CPD server lost connection with config server, the config server should unregister the daemon so that it can start up again, or auto re-connect (if using ADDP protocol).

If some unknown reason again, config server did not unregister it, then CPD server will not allow to start up.

Following config server log clearly shown that it though CPD server already running, and not allow the CPD which lost connectivity to it to reconnect agian


13:27:31.008 Trc 04541 Message MSGCFG_CLIENTREGISTER received from 1176 ( '')

MSGCFG_CLIENTREGISTER
  attr: SATRCFG_PROTOCOLEX          value: "CfgProtocol 5.1.3.75"
  attr: SATRCFG_PROTOCOL            value: "CfgProtocol 5.1.3.54"
  attr: SATRCFG_USERPASS            value: "******"
  attr: SATRCFG_USERNAME            value: ""
  attr: SATRCFG_APPPASS             value: ""
  attr: SATRCFG_APPNAME             value: "CPDNorth"
  attr: IATRCFG_APPTYPE             value: 0 [Unknown]
  attr: IATRCFG_REQUESTID           value: 366

14:27:31.008 Std 23500 Configuration Server Error : Error  [CFGAppSecurityViolation], object [CfgApplication], property [name] Description Server with this name is already running
14:11:31.008 Trc 04542 Message MSGCFG_ERROR sent to 1176 ( '')

Following CPD log showing it is trying to reconnect every 10 sec, and it is giving false alarm that config server not found, instead of process already started error found in config server log


13:27:52.517 Trc 11000 CfgServerReconnectTimer timer for 10 sec. was created: 0x5fe9160
Request ConfRestoreSessionAsync to restore connection to ConfigurationServer (300)
addp-trace off
(addp_xconfig) local OFF, remote OFF, trace off
Request to restore session failed (The specified Configuration Server cannot be found). Keep trying to restore session.
ConfigurationServer cannot be contacted.
Trying to restore session.
13:27:53.501 Trc 11000 CfgServerReconnectTimer timer for 10 sec. was created: 0x68a02c8
Request ConfRestoreSessionAsync to restore connection to ConfigurationServer (300)
addp-trace off
(addp_xconfig) local OFF, remote OFF, trace off
Request to restore session failed (The specified Configuration Server cannot be found). Keep trying to restore session.
ConfigurationServer cannot be contacted.
Trying to restore session.


Anyway, bouncing of CPD server will resolve this connectivity issue.

It looks like a bug between CPD and config server

Wireshark Capture Genesys T-Server SSL/TLS Network Packet

Product 1: Genesys CTI Suite (any version) for Windows
Product 2: Wireshark v1.6.4

There is nobody posted anything about capturing TLS encrypted network packet for Genesys CTI application, so I decided to write one.

This article contains configuration of Wireshark to capture SSL encrypted network packet communication with T-Server running on port 3000.

Overall activities are as below:
1. Export Windows SSL certificate as pfx file
2. Convert SSL certificate pfx file to pem
3. Capture network packet and restart application using Wireshark
4. Configure SSL pem certificate file

Step #3 is the most crucial step which capture the network packets. The rest of the steps can be done after the pcap file is saved. However, I am following above logical steps

Export SSL Certificate
Genesys uses Windows certificate feature, which is the same certificate used by IIS web  server. There are many references in Windows' SSL certificate export, so I will only show the major screen shots

Launch MMC and add snap-in Certificates. Alternatively, export it from IIS manager

Navigate to tree node Trusted Root Certification Authorities - Certificates. Look for the certificate name which match the T-Server hostname configured in Genesys Configuration Manager (CME). If there are several T-Server used, then export the SSL certificate from each of the server, and upload into a common directory. This will simplify the file gathering later

Choose output format as PKCS #12 (pfx). Ignore all the checkbox as Wireshark does not need any of those

Assume I have 2 T-Server, and saved them as tserver1.pfx, and tserver2.pfx.

Assume the pfx saved in C:\Wireshark\ directory

Convert pfx to pem Certificate
This step is not require for Windows, as Wireshark can accept pfx file. For other non-pfx certificate format, follow this section

Download OpenSSL for Windows, or Linux. This is a common utility comes pre-install for many Linux distribution. This free utility is provided by Shining Light Productions. Either 32-bit or 64-bit is fine, so for 64-bit Windows, either version will work.

http://www.slproweb.com/download/Win32OpenSSL-1_0_0e.exe (32-bit)

Program openssl.exe is the utility that needed to convert to pem certificate file.

Go to DOS prompt and execute following commands
cd C:\Wireshark

\Utilities\OpenSSL\bin\openssl pkcs12 -nodes -in tserver1.pfx -out tserver1.pem -nocerts -nodes

\Utilities\OpenSSL\bin\openssl pkcs12 -nodes -in tserver2.pfx -out tserver2.pem -nocerts -nodes

The content of the pem file will be similar to below


Bag Attributes
    Microsoft Local Key set:
    localKeyID: 01 00 00 00 
    Microsoft CSP Name: Microsoft RSA SChannel Cryptographic Provider
    friendlyName: c6bad00c9d00bfd55dc217383c14f1c5_d9350150-31e7-4e1f-889d-029377e717f2
Key Attributes
    X509v3 Key Usage: 10 
-----BEGIN RSA PRIVATE KEY-----
MIICXQIBAAKBgQC1PfnMaKjLpTyTZXW90FlLMYAeRsciKVpVjtx973gT6W552Tot
....cut...

FCR4IYW4+ye/IfbV4bYgDDyW4Wb1bk9bNnF6/U7pXJ3/
-----END RSA PRIVATE KEY-----

Capturing in Wireshark
In order to minimize the pcap network capture file, configure to capture only traffic sending to port 3000, which is the T-Server port for both T-Servers (or more)

Optionally, configured to save the captured data to file and auto rename file hourly. Following shown save to C:\Wireshark

Optionally, configured to stop capturing after 5 hours. This is handy if want to automatically stop network capturing, and eliminate filling up the disk space

Once the capturing start, restart any Genesys T-Server client, such as Siebel CRM, soft phone, OCM, CCPulse, StatServer, SAP soft phone, etc. This is because SSL certificate handshake mostly begin at the start of the application, or login. It contains crucial SSL encryption protocol which needed to decrypt the communication. If this step missed, then it is impossible to decode the network packet

During the capture, pay attention to specific application that needs to trace. For example, I select a CRM CTI application which communicate with T-Server on port 64406 (client) and 3000 (T-Server), which contains [PSH, ACK]. Wireshark will show an alias name stm-pproc by default but this is meaningless for Genesys T-Server troubleshooting

Right click on the packet, and choose decode as SSL

Click on Transport tab, and change TCP port to both. If the CTI application constantly change the port number, then only choose a packet sent from T-Server to client, and choose source port, and another packet sent from client to T-Server and choose destination port. In either case, choose protocol as SSL

The packet will change to TLSv1 protocol. In this case, it is a handshark fail, until we configured SSL pem certificate below. The SSL configuration can be done after the pcap file is saved. This is only for display purpose, mostly for real time analysis

Configure SSL pem Certificate
This step is required in order for Wireshark to show the encrypted TLS packet. As long as Wireshark capture the certificate handshake packets, it is able to decrypt the content after configure this.

Packet captured prior of SSL certificate configuration will still be able to be decrypted. This is a wrong understanding for many people (who do not understand SSL and TLS security).

Press Ctrl-Shift-P or from View - Preferences pull down menu to open the preference screen

Expand protocol on left panel, and scroll down to SSL

Click on RSA keys list Edit button, and fill in the pem certificate file for each T-Server. Click on Add button multiple time for each T-Server. Following screen shown T-Server 192.168.1.11 and .12 on port 3000 are added. Both pointing to different pem files

Once this step is done, previously SSL packet is readable

In above packets, we see following

1. Contain "Server Hello"
2. "Client Key Exchange" contain success message "Finished." It changed from "Encrypted Handshake Message" previously
3. "Change Cipher Spec" contains success message "Finished." It changed from "Encrypted Handshake Message" previously
4. Last packet changed from "Application Data" to "Continuation or non-HTTP traffic" with protocol as HTTP

Click on the HTTP with 141 bytes length show the decrypted content

It is found that pcap captured by Wireshark 1.6.4 is not able to decrypt by Wireshark 1.6.2. Not sure this is a known bug in Wireshark, but it does look like a bug to me.

Post your comment if you need help or clarification

Please use following PayPal donate if my post helped