Business Linux Operating Systems

Linux

Linux

Unix and Linux are different operating systems with have some common commands. Source code for Linux is freely available to the public and Unix is not available. Linux operating system is a free/open source and Some versions of Unix are proprietary and others are a free/open source. Linux Operating system can be used for desktop systems and for servers. But the Unix is mainly used in servers, mainframes and high-end computers.

AIX is an operating system based on Unix versions from IBM. It is mainly designed for IBM’s workstations and for the server hardware platforms. And HP-UX is the operating system from HP ( Hewlett Packard ) based on Unix versions.  HP-UX and AIX are stable operating system compare with Linux. HP-UX and AIX are platform dependent and they are limited to their own hardware. But in the case of Linux, it is platform independent and can be used with any hardware. Since HP-UX and AIX are platform dependent, they are optimised for the hardware and the performance is better than Linux operating systems.  AIX is outperforming Linux from 5 to 10 percent.

Unix

AT&T Unix, started in the 1970s at the Bell Labs and newer versions of Unix have developed and some of them are listed below. In 1980, AT&T licensed Unix to third-party vendors and leading to the development of different variants. Some of them are;

  • Berkeley Unix, FreeBSD and its variants
  • Solaris from Sun Microsystem
  • HP-UX from Hewlett-Packard
  • AIX from IBM
  • MacOs from Apple
  • Microsoft’s Xenix

Unix installations are costlier since it requires some special hardware. MacOS needs apple computers, AIX needs IBM hardware and HP-UX needs HP hardware etc.

Linux

Linux is a free and open source operating system based on Unix. Linux kernel was first developed by Linus Torvalds in 1991. Linux was originally developed for personal computers but nowadays it is using personal computers as well as in server systems. Since it is very flexible, it can be installed in any hardware systems. Linux operating system is available for mobile phones, tablets, video game consoles, mainframes and supercomputers. Some of the best distros for small business are;

  • Centos
  • ClearOS
  • OpenSUSE
  • IPFire
  • Ubuntu
  • Manjaro
  • Slackware

Linux Vs Unix

Linux Unix
The Source Code of Linux is freely available to its Users. The Source Code of Unix is not available for the general public.
Linux primarily uses Graphical User Interface with an optional Command Line Interface. Unix primarily uses Command Line Interface.
Linux OS is portable and can be executed in different Hard Drives. Unix is not portable.
Linux is very flexible and can be installed on most of the Home Based Pcs. Unix has a rigid requirement of the Hardware. Hence, cannot be installed on every other machine.
Linux is mainly used in Home Based PC, Mobile Phones, Desktops, etc. Unix is mainly used in Server Systems, Mainframes and High-End Computers.
Different Versions of Linux are: Ubuntu, Debian, OpenSuse, Redhat, Solaris, etc. Different Versions of Unix are: AIS, HP-UX, BSD, Iris, etc.
Linux Installation is economical and doesn’t require much specific and high-end hardware. Unix Installation is comparatively costlier as it requires more specific hardware circuitry.
The Filesystems supported by Linux are as follows: xfs, ramfs, nfs, vfat, cramfsm ext3, ext4, ext2, ext1, ufs, autofs, devpts, ntfs The Filesystems supported by Unix are as follows: zfs, js, hfx, gps, xfs, vxfs.
Linux is developed by an active Linux Community worldwide. Unix is developed by AT&T Developers.

Hardware architecture

Most commercial versions of UNIX distributions are coded for specific hardware. Like HP-UX for PA-RISC (Hewlett-Packard) and Itanium machines (Intel) and AIX is for Power processors ( IBM ). Since these distributions are limited, the developers can optimise their code for these architectures to get maximum utilisation of resources.  Since it uses proprietary hardware, Unix distributions are not cost effective.

  • HP-UX needs HP or Intel hardware
  • AIX needs IBM Hardware

Linux operating system is not dependent on the hardware, so it can be installed in any of the server systems which have a processor. Since the developers cannot assume the hardware architecture and they need to prepare the code for some general hardware specifications and that’s why Linux operating system has less performance than the commercial Unix variants.

  • Linux is open to all hardware

Licensing

GNU General Public License (GPL), is a form of copyleft and is used for the Linux kernel and many of the components from the GNU Project. Free software projects, although developed through collaboration, are often produced independently of each other. AIX and HP-UX are using proprietary licenses.

HP-UX

Developer Hewlett-Packard Enterprise
Written in C
OS family Unix (System V)
Initial release 1982; 36 years ago
Kernel type Monolithic with dynamically loadable modules
License Proprietary

 

IBM AIX

Developer IBM
Written in C
OS family Unix
Initial release 1986; 32 years ago
Kernel type Monolithic with dynamically loadable modules
License Proprietary

 

Linux

Developer Community, Linus Torvalds
Written in Primarily C and assembly
OS family Unix-like
Initial release September 17, 1991; 26 years ago
Kernel type Monolithic (Linux kernel)
License GPLv2[7] and other free and open-source licenses (the name “Linux” is a trademark[b])

 

Softwares and Tools

Softwares and tools in Linux are general to all hardware. But in the case of Unix, separate tools and software which leverage to get the maximum performance. So the performance of the systems is higher than the Linux operating system by comparing the hardware configuration. Unix has good performance than Linux systems. While considering the cost estimation, Linux will get more votes.

System Management Interface Tool ( SMIT ) with AIX is the tools used for package management, System Administration Manager (SAM) on HP-UX. Linux operating system uses rpm or dpkg etc. based on the variants.

Software Installation and Patch Management

R H Linux

HP-UX

AIX

Install rpm -i file swinstall –s depot software installp –a [-c] FileSet
Update rpm -U/F file swinstall –s depot software installp –a FileSet
List rpm -q swlist –l product lslpp –L all
Remove rpm -e swremove software installp –u FileSet
Patches rpm -u swinstall installp
List Patches rpm -q -a swlist –l product lslpp –L all
Patch check up2date/yum security_patch_check compare_report

File system

While talking about the file systems, Linux scores more than the other Unix versions. Unix supports two or three file systems locally. But Linux supports almost all the file systems available on any operating system.

 

System Filesystem
AIX jfs, gpfs
HP-UX hfs, vxfs

Kernel

The kernel is the core of the operating system and the source code of the kernel are not freely available for the commercial versions of Unix. For the Linux operating system, the users can check and verify the code and even modify it if required.

Support

The commercial versions of Unix come with a license cost. Since these operating systems are purchased, the vendor will provide technical support to the end users to the smooth running of the operating systems.

In the case of the Linux operating system, we need to use the open source forums and community for getting support from the users and developers around the world or hire some freelancers for fixing the issues.

Related References

Big Data vs. Virtualization

Big Data Information Approaches

Big Data Information Approaches

Globally, organizations are facing challenges emanating from data issues, including data consolidation, value, heterogeneity, and quality. At the same time, they have to deal with the aspect of Big Data. In other words, consolidating, organizing, and realizing the value of data in an organization has been a challenge over the years. To overcome these challenges, a series of strategies have been devised. For instance, organizations are actively leveraging on methods such as Data Warehouses, Data Marts, and Data Stores to meet their data assets requirements. Unfortunately, the time and resources required to deliver value using these legacy methods is a distressing issue. In most cases, typical Data Warehouses applied for business intelligence (BI) rely on batch processing to consolidate and present data assets. This traditional approach is affected by the latency of information.

Big Data

As the name suggests, Big Data describes a large volume of data that can either be structured or unstructured. It originates from business processes among other sources. Presently, artificial intelligence, mobile technology, social media, and the Internet of Things (IoT) have become new sources of vast amounts of data. In Big Data, the organization and consolidation matter more than the volume of the data. Ultimately, big data can be analyzed to generate insights that can be crucial in strategic decision making for a business.

Features of Big Data

The term Big Data is relatively new. However, the process of collecting and preserving vast amounts of information for different purposes has been there for decades. Big Data gained momentum recently with the three V’s features that include volume, velocity, and variety.

Volume: First, businesses gather information from a set of sources, such as social media, day-to-day operations, machine to machine data, weblogs, sensors, and so on. Traditionally, storing the data was a challenge. However, the requirement has been made possible by new technologies such as Hadoop.

Velocity: Another defining nature of Big Data is that it flows at an unprecedented rate that requires real-time processing. Organizations are gathering information from RFID tags, sensors, and other objects that need timely processing of data torrents.

Variety: In modern enterprises, information comes in different formats. For instance, a firm can gather numeric and structured data from traditional databases as well as unstructured emails, video, audio, business transactions, and texts.

Complexity: As mentioned above, Big Data comes from diverse sources and in varying formats. In effect, it becomes a challenge to consolidate, match, link, cleanse, or modify this data across an organizational system. Unfortunately, Big Data opportunities can only be explored when an organization successfully correlates relationships and connects multiple data sets to prevent it from spiraling out of control.

Variability: Big Data can have inconsistent flows within periodic peaks. For instance, in social media, a topic can be trending, which can tremendously increase collected data. Variability is also common while dealing with unstructured data.

Big Data Potential and Importance

The vast amount of data collected and preserved on a global scale will keep growing. This fact implies that there is more potential to generate crucial insights from this information. Unfortunately, due to various issues, only a small fraction of this data actually gets analyzed. There is a significant and untapped potential that businesses can explore to make proper and beneficial use of this information.

Analyzing Big Data allows businesses to make timely and effective decisions using raw data. In reality, organizations can gather data from diverse sources and process it to develop insights that can aid in reducing operational costs, production time, innovating new products, and making smarter decisions. Such benefits can be achieved when enterprises combine Big Data with analytic techniques, such as text analytics, predictive analytics, machine learning, natural language processing, data mining and so on.

Big Data Application Areas

Practically, Big Data can be used in nearly all industries. In the financial sector, a significant amount of data is gathered from diverse sources, which requires banks and insurance companies to innovate ways to manage Big Data. This industry aims at understanding and satisfying their customers while meeting regulatory compliance and preventing fraud. In effect, banks can exploit Big Data using advanced analytics to generate insights required to make smart decisions.

In the education sector, Big Data can be employed to make vital improvements on school systems, quality of education and curriculums. For instance, Big Data can be analyzed to assess students’ progress and to design support systems for professors and tutors.

Healthcare providers, on the other hand, collect patients’ records and design various treatment plans. In the healthcare sector, practitioners and service providers are required to offer accurate and timely treatment that is transparent to meet the stringent regulations in the industry and to enhance the quality of life. In this case, Big Data can be managed to uncover insights that can be used to improve the quality of service.

Governments and different authorities can apply analytics to Big Data to create the understanding required to manage social utilities and to develop solutions necessary to solve common problems, such as city congestion, crime, and drug use. However, governments must also consider other issues such as privacy and confidentiality while dealing with Big Data.

In manufacturing and processing, Big Data offers insights that stakeholders can use to efficiently use raw materials to output quality products. Manufacturers can perform analytics on big data to generate ideas that can be used to increase market share, enhance safety, minimize wastage, and solve other challenges faster.

In the retail sector, companies rely heavily on customer loyalty to maintain market share in a highly competitive market. In this case, managing big data can help retailers to understand the best methods to utilize in marketing their products to existing and potential consumers, and also to sustain relationships.

Challenges Handling Big Data

With the introduction of Big Data, the challenge of consolidating and creating value on data assets becomes magnified. Today, organizations are expected to handle increased data velocity, variety, and volume. It is now a business necessity to deal with traditional enterprise data and Big Data. Traditional relational databases are suitable for storing, processing, and managing low-latency data. Big Data has increased volume, variety, and velocity, making it difficult for legacy database systems to efficiently handle it.

Failing to act on this challenge implies that enterprises cannot tap the opportunities presented by data generated from diverse sources, such as machine sensors, weblogs, social media, and so on. On the contrary, organizations that will explore Big Data capabilities amidst its challenges will remain competitive. It is necessary for businesses to integrate diverse systems with Big Data platforms in a meaningful manner, as heterogeneity of data environments continue to increase.

Virtualization

Virtualization involves turning physical computing resources, such as databases and servers into multiple systems. The concept consists of making the function of an IT resource simulated in software, making it identical to the corresponding physical object. Virtualization technique uses abstraction to create a software application to appear and operate like hardware to provide a series of benefits ranging from flexibility, scalability, performance, and reliability.

Typically, virtualization is made possible using virtual machines (VMs) implemented in microprocessors with necessary hardware support and OS-level implementations to enhance computational productivity. VMs offers additional convenience, security, and integrity with little resource overhead.

Benefits of Virtualization

Achieving the economics of wide-scale functional virtualization using available technologies is easy to improve reliability by employing virtualization offered by cloud service providers on fully redundant and standby basis. Traditionally, organizations would deploy several services to operate at a fraction of their capacity to meet increased processing and storage demands. These requirements resulted in increased operating costs and inefficiencies. With the introduction of virtualization, the software can be used to simulate functionalities of hardware. In effect, businesses can outstandingly eliminate the possibility of system failures. At the same time, the technology significantly reduces capital expense components of IT budgets. In future, more resources will be spent on operating, than acquisition expenses. Company funds will be channeled to service providers instead of purchasing expensive equipment and hiring local personnel.

Overall, virtualization enables IT functions across business divisions and industries to be performed more efficiently, flexibly, inexpensively, and productively. The technology meaningfully eliminates expensive traditional implementations.

Apart from reducing capital and operating costs for organizations, virtualization minimizes and eliminates downtime. It also increases IT productivity, responsiveness, and agility. The technology provides faster provisioning of resources and applications. In case of incidents, virtualization allows fast disaster recovery that maintains business continuity.

Types of Virtualization

There are various types of virtualization, such as a server, network, and desktop virtualization.

In server virtualization, more than one operating system runs on a single physical server to increase IT efficiency, reduce costs, achieve timely workload deployment, improve availability and enhance performance.

Network virtualization involves reproducing a physical network to allow applications to run on a virtual system. This type of virtualization provides operational benefits and hardware independence.

In desktop virtualization, desktops and applications are virtualized and delivered to different divisions and branches in a company. Desktop virtualization supports outsourced, offshore, and mobile workers who can access simulate desktop on tablets and iPads.

Characteristics of Virtualization

Some of the features of virtualization that support the efficiency and performance of the technology include:

Partitioning: In virtualization, several applications, database systems, and operating systems are supported by a single physical system since the technology allows partitioning of limited IT resources.

Isolation: Virtual machines can be isolated from the physical systems hosting them. In effect, if a single virtual instance breaks down, the other machine, as well as the host hardware components, will not be affected.

Encapsulation: A virtual machine can be presented as a single file while abstracting other features. This makes it possible for users to identify the VM based on a role it plays.

Data Virtualization – A Solution for Big Data Challenges

Virtualization can be viewed as a strategy that helps derive information value when needed. The technology can be used to add a level of efficiency that makes big data applications a reality. To enjoy the benefits of big data, organizations need to abstract data from different reinforcements. In other words, virtualization can be deployed to provide partitioning, encapsulation, and isolation that abstracts the complexities of Big Data stores to make it easy to integrate data from multiple stores with other data from systems used in an enterprise.

Virtualization enables ease of access to Big Data. The two technologies can be combined and configured using the software. As a result, the approach makes it possible to present an extensive collection of disassociated and structured and unstructured data ranging from application and weblogs, operating system configuration, network flows, security events, to storage metrics.

Virtualization improves storage and analysis capabilities on Big Data. As mentioned earlier, the current traditional relational databases are incapable of addressing growing needs inherent to Big Data. Today, there is an increase in special purpose applications for processing varied and unstructured big data. The tools can be used to extract value from Big Data efficiently while minimizing unnecessary data replication. Virtualization tools also make it possible for enterprises to access numerous data sources by integrating them with legacy relational data centers, data warehouses, and other files that can be used in business intelligence. Ultimately, companies can deploy virtualization to achieve a reliable way to handle complexity, volume, and heterogeneity of information collected from diverse sources. The integrated solutions will also meet other business needs for near-real-time information processing and agility.

In conclusion, it is evident that the value of Big Data comes from processing information gathered from diverse sources in an enterprise. Virtualizing big data offers numerous benefits that cannot be realized while using physical infrastructure and traditional database systems. It provides simplification of Big Data infrastructure that reduces operational costs and time to results. Shortly, Big Data use cares will shift from theoretical possibilities to multiple use patterns that feature powerful analytics and affordable archival of vast datasets. Virtualization will be crucial in exploiting Big Data presented as abstracted data services.

 

End Of Support For IBM InfoSphere 9.1.0

IBM Information Server (IIS)

IBM Information Server (IIS)

End of Support for IBM InfoSphere Information Server 9.1.0

IBM InfoSphere Information Server 9.1.0 will reach End of Support on 2018-09-30.  If you are still on the InfoSphere Information Server (IIS) 9.1.0, I hope you have a plan to migrate to an 11-series version soon.  InfoSphere Information Server (IIS) 11.7 would be worth considering if you don’t already own an 11-series license. InfoSphere Information Server (IIS) 11.7 will allow you to take advantage of the evolving thin client tools and other capabilities in the 2018 release pipeline without needing to perform another upgrade.

Related References

IBM Support, End of support notification: InfoSphere Information Server 9.1.0

IBM Support, Software lifecycle, InfoSphere Information Server 9.1.0

IBM Knowledge Center, Home, InfoSphere Information Server 11.7.0, IBM InfoSphere Information Server Version 11.7.0 documentation

My Most Used Windows 10 Keyboard Shortcuts

Shortcut Keystrokes

Shortcut Keystrokes

While there are a great number of useful windows 10 shortcuts, I have the list below the combination, which I use daily.  Many of the shortcuts can be used across multiple applications (e.g. Notepad++, MS Word, SQL Server, Aginity, etc.) and save a considerable amount of mouse work.  Overall, these shortcut keys are more efficient and faster than using the mouse to perform the same task on a repetitive basis.

You may want to investigate the numerous other Windows 10 shortcuts keys, which best apply to your daily activities, but these are the ones, which I have found most useful and which I have committed to memory.

Table of My Most Used Windows Shortcuts

Key
Strokes

Behavior

Alt
+ Tab

Switch
between open apps

Ctrl
+ A

Select
all items in a document or window

Ctrl
+ Alt + Tab

Use
the arrow keys to switch between all open apps

Ctrl
+ C

Copy
the selected item

Ctrl
+ D

Delete
the selected item and move it to the Recycle Bin

Ctrl
+ F

Select
the search box

Ctrl
+ V

Paste
the selected item

Ctrl
+ X

Cut
the selected item

Esc

Stop
or leave the current task

F5

Refresh
the active window

F11

Maximize
Window

Related References

 Microsoft > Windows Support > Keyboard shortcuts in Windows

 

 

 

 

 

 

Parallel jobs on Windows fail with APT_IOPort::readBlkVirt;error

APT_IOPort::readBlkVirt Error Screenshot

APT_IOPort::readBlkVirt Error Screenshot

This a known error for windows systems and applies to DataStage and DataQuality jobs using the any or all the three join type stages (Join, Merge, and Lookup).

Error Message

  • <<Link name>>,0: APT_IOPort::readBlkVirt: read for block header, partition 0, [fd 4], returned -1 with errno 10,054 (Unknown error)

Message ID

  • IIS-DSEE-TFIO-00223

Applies To

  • Windows systems only
  • Parallel Engine Jobs the three join type stages (Join, Merge, and Lookup). It does not apply to Server Engine jobs.
  • Infosphere Information Server (IIS), Datastage and DataQuality 9.1 and higher

The Fix

  • Add the APT_NO_IOCOMM_OPTIMIZATION in project administrator and set to blank or 0. I left it blank so it would not impact other jobs
  • Add the environment variable to the job producing the error and set to 1

What it APT_NO_IOCOMM_OPTIMIZATION Does

  • Sets the use of shared memory as the transport type, rather than using the default sockets transport type.
  • Note that in most cases sockets transport type is faster, so, you likely will not to set this across the project as the default for all job. It is best to apply it as necessary for problematic jobs.

Related References

InfoSphere DataStage and QualityStage, Version 9.1 Job Compatibility

IBM Support, JR54078: PARALLEL JOBS ON WINDOWS FAIL WITH APT_IOPORT::READBLKVIRT; ERROR

IBM Support, Information Server DataStage job fails with unknown error 10,054.

 

How to know if your Oracle Client install is 32 Bit or 64 Bit

Oracle Database, How to know if your Oracle Client install is 32 Bit or 64 Bit

Oracle Database

 

How to know if your Oracle Client install is 32 Bit or 64 Bit

Sometimes you just need to know if your Oracle Client install is 32 bit or 64 bit. But how do you figure that out? Here are two methods you can try.

The first method

Go to the %ORACLE_HOME%\inventory\ContentsXML folder and open the comps.xml file.
Look for <DEP_LIST> on the ~second screen.

If you see this: PLAT=”NT_AMD64” then your Oracle Home is 64 bit
If you see this: PLAT=”NT_X86” then your Oracle Home is 32 bit.

It is possible to have both the 32-bit and the 64-bit Oracle Homes installed.

The second method

This method is a bit faster. Windows has a different lib directory for 32-bit and 64-bit software. If you look under the ORACLE_HOME folder if you see a “lib” AND a “lib32” folder you have a 64 bit Oracle Client. If you see just the “lib” folder you’ve got a 32 bit Oracle Client.

Related References

 

Oracle – How to get a list of user permission grants

IBM Infosphere Information Server (IIS), Oracle – How to get a list of user permission grants

IBM Infosphere Information Server (IIS)

Since the Infosphere, information server, repository, has to be installed manually with the scripts provided in the IBM software, sometimes you run into difficulties. So, here’s a quick script, which I have found useful in the past to identify user permissions for the IAUSER on Oracle database’s to help rundown discrepancies in user permissions.

 

SELECT *

FROM ALL_TAB_PRIVS

WHERE  GRANTEE = ‘iauser’

 

If we cannot run against the ALL_TAB_PRIVS view, then we can try the ALL_TAB_PRIVS view:

 

SELECT *

FROM USER_TAB_PRIVS

WHERE  GRANTEE = ‘iauser’

 

Related References

oracle help Center > Database Reference > ALL_TAB_PRIVS view

Linux – What is yum?

Linux

Linux

In simple terms, yum is a, command-line interface, package manager utility for computers running the Linux operating system, which augments the RPM Package Manager capabilities. yum is the primary tool for getting, installing, deleting, querying, and managing RPM software packages. Alos, yum is used in Red Hat Enterprise Linux (RHEL) versions 5 and later.

 

Where to download IBM Data Studio?

IBM Data Studio Client

IBM Data Studio Client

IBM data studio is offered free from IBM, and can be helpful when working with DB2 and Puredata/Netezza using a JDBC driver.

What you need to Down Load IBM Data Studio

  • You will need an IBM ID and password

Basic down load steps

IBM Sign In Screen

IBM Sign In Screen

  • Enter you IBM ID, and password, then click ‘sign in’.
  • On the IBM Data Studio Client, license page, check ‘I agree’ and then click ‘I confirm’
IBM Data Studio Client License Screen

IBM Data Studio Client License Screen

  • On the IBM Data Studio Client, download page, Select the desired method tab, Then
    • Select the desired product or products and click ‘Download now.
IBM Data Studio Client Download Files Screen

IBM Data Studio Client Download Files Screen

 

Related References

IBM Data Studio

IBM Software > Products > Data management platform > Data management > IBM Data Studio

IBM Data Studio Client (Download)

IBM Support

Download and install IBM Data Studio Version 4.1.x

IBM Support

System requirements for IBM Data Studio Version 4.1.x

IBM Knowledge Center

Data Studio, Data Studio 4.1.1, Overview, Overview of IBM Data Studio

 

Linux – how to display file system disk space statistics

Linux

Linux

In Linux there are lot of ways to disk size and/or space, but the ‘Disk Filesystem’ (df) command is old reliable and has been around a long time.   The ‘df’ command provides a summary of disk space and free space, which I find myself coming back to time after time.

Basic Command Format

DF -<<Option>>   <<File>>

Example ‘Disk Filesystem’, Command

df -h

  • -h = Human readable in Mega Bytes

For more details in Linux

df –help

 

Example Command Output

root@BlogSrvr1 /]# df -h

Filesystem            Size  Used Avail Use% Mounted on

/dev/mapper/vg_BlogSrvr1-lv_root

36G   34G   16M 100% /

tmpfs                 3.9G     0  3.9G   0% /dev/shm

/dev/sda1             477M   33M  419M   8% /boot

/dev/mapper/vg_BlogSrvr1-LogVol03

11G   27M  9.9G   1% /data

/dev/mapper/vg_BloSrvr1-lv_home

4.8G   33M  4.6G   1% /home

/dev/mapper/vg_BlogSrvr1-LogVol04

25G   13G   11G  55% /opt/IBM

/dev/mapper/vg_BlogSrvr1-LogVol05

11G  6.0G  3.7G  62% /scratch

/dev/mapper/vg_BlogSrvr1-LogVol06

11G   27M  9.9G   1% /tmp/dev/shm

*DataStage*DSR_SELECT (Action=3); check DataStage is set up correctly in project

Error

Error

Having encountered this DataStage client error in Linux a few times recently, I thought I would document the solution, which has worked for me.

Error Message:

Error calling subroutine: *DataStage*DSR_SELECT (Action=3); check DataStage is set up correctly in project

(Subroutine failed to complete successfully (30107))

Probable Cause of Error

  • NodeAgents has stopped running
  • Insufficient /temp disk space

Triage Approach

To fix this error in Linux:

  • Ensure disk space is available and you may want clean up the /tmp directory of any excel non-required files.
  • Start the NodeAgents.sh, if it is not running

Command to verify Node Agent is running

ps -ef | grep java | grep Agent

 

Command to Start Node Agent

This example command assumes the shell script is in its normal location, if not you will need to adjust the path.

/opt/IBM/InformationServer/ASBNode/bin/NodeAgents.sh start

Node Agent Logs

These logs may be helpful:

  • asbagent_startup.err
  • asbagent_startup.out

Node Agent Logs Location

This command will get you to where the logs are normally located:

cd /opt/IBM/InformationServer/ASBNode/

Linux – How to compress an entire directory

Linux

Linux

From time to time there is a need to package up a folder for any number of reasons which may include things like:

  • Migration
  • Movement to a new location
  • Movement to a new server
  • To keep a backup
  • Or simply to save space

Compressing a folder is folder can be very useful, but for those of us who don’t do it all the time, it is nice to have a pattern to follow.  Also, even an experienced user can get brain cramp, if they have not had a reason to compress a folder in a while. So, here is a simple pattern to follow to compress a folder and its contents.

Basic Command Format

tar -zcvf <<archive-name>>.tar.gz <<directory-name>>

Example Compress Command

tar -zcvf  blog_files_backup.tar.gz   sqlfiles

Linux tar command line options used here

  • -z = Compress archive using gzip program
  • -c = Create archive
  • -v = Verbose i.e display progress while creating archive
  • -f = Archive File name

For help with the tar command in Linux

To get additional detail on the tar command in Linux, just need to type:

 tar -?

 

Netezza – [SQLCODE=HY000][Native=46] ERROR: External Table : count of bad input rows reached maxerrors limit

SQL (Structured Query Language)

SQL (Structured Query Language)

While helping a customer we encountered the [SQLCODE=HY000][Native=46] ERROR, which was a new one for me. So here are a few notes to help the next unlucky soul may run into the error.

Netezza Error Reason:

  • [SQLCODE=HY008][Native=51] Operation canceled; [SQLCODE=HY000][Native=46] ERROR: External Table : count of bad input rows reached maxerrors limit

What Does the Error Mean

  • In a nutshell, it mean invalid data was submitted and could not be inserted.

What To Do

  • Basically, you need to go to the Netezza logs to see why the rows were reject and resolve input data error, then resubmit your transactions. The logs are temporary and reused, so, you need to get to them before they are over written.

Where Are The Data Logs

  • In linux the logs can be found in /tmp:

For nzload Methods Logs

  • /tmp/database name.table name.nzlog
  • /tmp/database name.table name.nzbad

For External Table Load Logs

  • /tmp/external table name.log
  • /tmp/external table name.bad

Related References

 

How to stop and restart Cognos Service from Linux command line

stop and restart cognos service from linux command line

stop and restart cognos service from linux command line

I don’t do this very often, but recent had to look this up to help out a project.  Stopping and restarting a Cognos from a Linux command line is relatively simple, just a couple of commands.

  • Log on to the reporting server as Root user or a non-root user with administrative privileges.
  • Find the path to install bin directory.  I use this find command, but you can do what works for you:   find . -name “cogconfig.sh”
  • Launch an and navigate to the bin directory as follows: <Cognos_Home>/bin64
  • Where <Cognos_Home> is the installation location of the Cognos® application.
  • Do the following one or both of the following, according to what you are attempting to do:
    • To start the service, enter the following command: ./cogconfig.sh -s
    • To stop the service, enter the following command: ./cogconfig.sh -stop

 

Related References

 

Surrogate Key File Effective Practices

Database Table, Surrogate Key File Effective Practices

Surrogate Key File Effective Practices

Here are a few thoughts on effectively working with IBM Infosphere, Information Server, DataStage surrogate key files, which may prove useful for developers.

 

Placement

  • The main thing about placement is that it be in a consistent location. Developers and production support teams should need to guess or look up where it is for every DataStage project. So, it is best to put the surrogate keys in same base path and that each project has its own subfolder to facilitate migrations and to reduce the possibility of human error. Here Is the patch structure, which is commonly use:

Path

  • /data/SRKY/<<Project Name>>

Parameter Sets

  • As a best practice, the surrogate key file path should be in a parameter set and the parameter used in the jobs, as needed.  This simplifies maintenance, if and when changes to the path are required, and during migrations.

Surrogate Key Parameter Set Screenshot – Example Parameter Tab

Surrogate Key Parameter Set Screen Screen

Surrogate Key Parameter Set Screenshot

Surrogate Key Parameter Set Screenshot – Example Values Tab

Surrogate Key Parameter Set Screenshot – Example Values Tab

Surrogate Key Parameter Set Screenshot – Example Values Tab

Surrogate Key Job Parameter Example Path using Parameter

Surrogate Key Job Parameter Example Path using Parameter

Surrogate Key Job Parameter Example Path using Parameter

Permissions

  • DataStage must have permissions to:
    • The entire parent path
    • The project folder, and
    • The surrogate key files themselves.

To ensure the DataStage has access to the path and Surrogate files, ensure:

  • The ‘dsadm’ (owner) and ‘dstage’ (group) have access to folders in the path, with at least a “-rw-r–r–“ (644) permissions level. Keeping the permissions to a minimum can, for obvious reasons,  prevent inadvertent overwrites of the surrogate key files; thus, avoiding some, potentially, serious cleanup.
  • The ‘dsadm’ (owner) and ‘dstage’ (group) have access to the surrogate key files

Surrogate Key File Minimum Permissions

Surrogate Key File Minimum Permissions

Surrogate Key File Minimum Permissions

Productivity Tip – Quickly create a new surrogate key file

Linux

Linux

This productivity tip, is how we can quickly create a new surrogate key file in Linux.  This example is leveraging native capabilities of Red Hat Enterprise Linux (RHEL) to skip a few commands, by using an existing surrogate key file to create a new surrogate file with a minimum of keys strokes and command line entries.

Creating a New Surrogate Key File From an Existing File

The basic process consists of just a few steps:

  1. Navigate to the location of your existing surrogate key files
  2. Copy an existing surrogate file
  3. Empty the new surrogate key file

Navigate to the location of your existing surrogate key files

This step is preparatory step; you will need to look at the path variable for the project you are working with to know where to go.  The actual path to the surrogate files your project can vary by project.

Copy an existing surrogate file

Assuming you have existing surrogate key files configured as needed, the use of the copy (cp) command can and the interactive and preserve options can eliminate the need to create the file, then set groups and permissions.   The interactive (-i) option prevent you from overwriting an existing files, in case you made a filename typo and the preserver (-p) option preserve the specified attributes (e.g. ownership, and permissions).

Basic Command

  • Here is the command formats with interactive and preserve, either format works
    • cp -ip <<FileName to Be Copied>> <<New Filename>>
  • Here is the command formats with only preserve
    • cp -p <<FileName to Be Copied>> <<New Filename>>

Example Command

  • cp -ip srky  blogexampl.srky
Copy Surrogate Key With Permissions

Copy Surrogate Key With Permissions

Empty the new surrogate key file

Setting the newly create surrogate key file to null will empty the file, so, DataStage can begin from the point configure in your DataStage job.

Basic Command

  • cat /dev/null > <<FileName>>

Example Command

  • cat /dev/null > blogexample.srky
Empty Surrogate Key File

Empty Surrogate Key File

Related References

 

Productivity Tip – Changing Owner and Groups on Surrogate Key File

Linux

Linux

 

This practice tip, is how we quickly update surrogate key file owner and group in Linux.  This example is leveraging native capabilities of Red Hat Enterprise Linux (RHEL) to skip a few commands, by using a combined command to set both the owner and group of a surrogate key file in a single command.

Surrogate Key File Owners and Groups

To ensure the DataStage has access to the path and Surrogate files, ensure the ‘dsadm’ (owner) and ‘dstage’ (group) have access to the surrogate key files

Setting Surrogate Key File Owners and Groups

You can change the ownership and group of a surrogate file at the same time, in Linux, with the change owner command. To do this navigate the surrogate key path containing the file, then execute the chown combined command.

Command chown basic format

  • chown <<OWNER>>:<<Group>> <<File Name>>

Example chown command

  • chown dsadm:dstage Blogexampl.txt
Chown On Surrogate Key File

Chown On Surrogate Key File

Related Reference

 

 

Infosphere Datastage – Client Tier Image Requirements

InfoSphere Information Server (IIS) DataStage Client Tools

InfoSphere Information Server (IIS) DataStage Client Tools

A frequent question encountered in the early stages of a client engagement is: what are the recommended Windows Client Tier Image Requirements (Virtual machine image or Desktop)?  However, I have never found this information to be satisfactorily documented by IBM.  So, invariably, we end up providing our best guidance based on experience, which in the case of the normal InfoSphere Information Server (IIS) installation is provided in the table below.

Recommended Developer Client Tier Characteristics

Item Quantity Notes

Application Directory Storage

8 GB or Larger

Memory

6 GB or More This should be a developer images, so, more is better.  Especially, given the other applications, which the developer may also be using simultaneously.
CPU Cores 2 or more This should be a developer images, so, more is better.  Especially, given the other applications, which the developer may also be using simultaneously.

Permissions

N/A Users performing the client software installation, must have full Administrative rights on the Virtual Machine image or individual workstation.

Protection Software

N/A All Firewalls and virus protection software needs to be disabled on the client, while client software installation is in progress.  Also, required Infosphere firewall ports for the client tools must be open to use the tools.

 

Related References

Are the Infosphere Windows Client 32 bit or 64 bit?

InfoSphere Information Server (IIS) DataStage Client Tools

InfoSphere Information Server (IIS) DataStage Client Tools

The question of whether or not the IBM InfoSphere Windows client tools 32 bit or 64 bit, actually, comes up rather frequently. The short answer to the question is that the InfoSphere Windows client tools, actually, are 32 bit applications, will run on supported 64 bit windows system.  This is what IBM calls 64-Tolerate: (32-bit applications that can run in a 64-bit operating system environment), however, the client tools do not run in native 64-bit mode and do not exploit the 64-bit operating environment to improve performance.

Related References

 

Netezza / PureData – How to find and kill table locks

Netezza/PureData - How to find and kill table locks

Netezza/PureData Table Session Locks

Sometimes there is a need to find and/or kill (terminate) table locks, so, that application process and user access can be restored.  To do this relative straight forward if you have access and the appropriate permission to the Netezza PureData server.

How to find table locks on a Netezza database

  • Log into the Netezza server
  • From the command line, navigate to Netezza directory (e.g. cd /NZ)
  • On the command line enter show locks command

 

Show Locks Command (nz_show_locks) Syntax

nz_show_locks <db name> <tablename>

 

Example Show Locks Command (nz_show_locks)

nz_show_locks dashboard_staging stg_nz_query_history

Netezza PureData Kill Table Session locks

Netezza PureData Kill Table Session locks

How to kill table locks on a Netezza database

  • Perform find locks above
  • Then, on the command line enter kill sessions locks command

Kill Sessions Command (nzsession) Syntax

nzsession subcmd [subcmd options]

Example Kill Sessions Command (nzsession)

nzsession abort -id  523662 -force

Related References:

 

l