Thursday, December 2, 2010

MySQL Database Optimization

This is specially tested with Snort IDS Database

Select your database and then issue the command below, where 'mytablename' is the name of the table you want to query
show table status like 'mytablename'\G
You can omit the "like 'mytablename'" part and then it will show this information for all tables.
However if you have a lot of tables and there's only one or two you want to examine then it's better to specify the particular table.
You can end you query with either ; or \G. I prefer \G for this particular query because it shows each column from the resultset on a new line,
whereas ; will show the columns across the screen. This is OK for a resultset with only a few columns with only a
small amount of information in each one, but it's not so good for this query.

The result from the above will look something like so:

*************************** 1. row ***************************
Name: mytablename
Engine: MyISAM
Version: 10
Row_format: Dynamic
Rows: 2444
Avg_row_length: 7536
Data_length: 564614700
Max_data_length: 281474976710655
Index_length: 7218176
Data_free: 546194608
Auto_increment: 1187455
Create_time: 2008-03-19 10:33:13
Update_time: 2008-09-02 22:18:15
Check_time: 2008-08-27 23:07:48
Collation: latin1_swedish_ci
Checksum: NULL
Create_options: pack_keys=0
Comment:
***************************************************************

The values that are important for working out if the table is non optimal is the "Data_free" value.
If this is high, as in the above example where 564614700 bytes are free (538MB),
then the table has a lot of space not being used and should be optimized.
To optimize the table, issue the following command, where "mytablename" is the name of the MySQL table to optimise:

optimize table mytablename;

After doing this (it may take a few seconds dpending on the size of the table, free space etc) and running "show table status" again, the result should look much better:
*************************** 1. row ***************************
Name: tblmailqueue
Engine: MyISAM
Version: 10
Row_format: Dynamic
Rows: 6145
Avg_row_length: 7505
Data_length: 46119636
Max_data_length: 281474976710655
Index_length: 296960
Data_free: 0
Auto_increment: 1191156
Create_time: 2008-03-19 10:33:13
Update_time: 2008-09-02 22:24:58
Check_time: 2008-09-02 22:21:32
Collation: latin1_swedish_ci
Checksum: NULL
Create_options: pack_keys=0
Comment:
1 row in set (0.00 sec)
***************************************************************
In the above example we can see the "Data_free" value is now zero so the table is nicely optimised.

You can do something like this:
SELECT concat("OPTIMIZE TABLE ", table_schema,".",table_name,";") FROM tables WHERE DATA_FREE > 0 INTO OUTFILE '/tmp/optimize.sql'; SOURCE '/tmp/optimize.sql';

SQL SERVER DATABASE MAINTENANACE - BEST PRACTICES BY MICROSOFT

Data-file (MDF and LDF) Maintenance

Transaction Log grows unexpectedly or becomes full on a SQL Server >> http://support.microsoft.com/?id=317375
How to move SQL Server databases to a new location http://support.microsoft.com/kb/224071
How to move databases between computers that are running SQL Server http://support.microsoft.com/kb/314546

DATABASE MaintenanceDatabase Maintenance >> http://technet.microsoft.com/en-us/magazine/2008.08.database.aspx
How to Minimize Deadlocks in SQL Server >> http://msdn.microsoft.com/en-us/library/ms191242(SQL.90).aspx
How to Minimize Blocking in SQL Server >> http://technet.microsoft.com/en-us/magazine/2008.04.blocking.aspx

SQL SERVER TRANSACTION LOG MAINTENANACE - BEST PRACTICES

A transaction log grows unexpectedly or becomes full on a computer that is running SQL Server http://support.microsoft.com/kb/317375
How to stop the transaction log of a SQL Server database from growing unexpectedly http://support.microsoft.com/kb/873235
Shrinking a Database http://msdn.microsoft.com/en-us/library/aa933076(SQL.80).aspx

MS SQL Database Full Recovery from the TRANSACTION LOGS ( Tested with MsSQL - 2005, 2008 )

MsSQL Database full Recovery from transaction logs

Spl.Note :-
I am assuming that your Database recovery mode is set to FULL and not the SIMPLE.
You have a database which is set to FULL recovery and u have a full Database backup on say Dec 2009 after
that u have regularly backed up its transactional logs and suddenly your database crashed or by any chance it is being altered.,

in this scenario u need to get the changes made to your database on a previous day.

in this conditions you must be having database complete backup which is taken on Dec 2009 or any latest full database backup and transactional logs till today, in our need we assume that the database restore requirement is till yesterday and not today as i discussed above.

Now its time to start the Restore Database operations.

Step:-1
RESTORE DATABASE YOUR_DATABASE_NAME
FROM DISK = 'D:\YOUR_DATABASE_PATH\YOUR_FULL_DATABASE_BACKUP.bak'
WITH NORECOVERY

The above command will results in following output

Processed 184 pages for database 'YOUR_FULL_DATABASE_BACKUP', file 'YOUR_FULL_DATABASE_BACKUP' on file 1.
Processed 6 pages for database 'YOUR_FULL_DATABASE_BACKUP', file 'YOUR_FULL_DATABASE_BACKUP_log' on file 1.
RESTORE DATABASE successfully processed 190 pages in 0.205 seconds (7.233 MB/sec).

Once you restore the full backup using the NORECOVERY option, you can begin applying the transaction log backups or the differential backup as given below.

Step:-2

RESTORE LOG YOUR_DATABASE_NAME
FROM DISK = 'D:\YOUR_DATABASE_PATH\Transaction_Logs_1.trn' WITH NORECOVERY

The above command will results in following output
Processed 0 pages for database 'YOUR_DATABASE_NAME', file 'YOUR_DATABASE_NAME' on file 1.
Processed 7 pages for database 'YOUR_DATABASE_NAME', file 'YOUR_DATABASE_NAME_log' on file 1.
RESTORE LOG successfully processed 7 pages in 0.017 seconds (2.900 MB/sec).

Step:-3
RESTORE LOG YOUR_DATABASE_NAME
FROM DISK = 'D:\YOUR_DATABASE_PATH\Transaction_Logs_2.trn' WITH NORECOVERY

The above command will results in following output
Processed 0 pages for database 'YOUR_DATABASE_NAME', file 'YOUR_DATABASE_NAME' on file 1.
Processed 3 pages for database 'YOUR_DATABASE_NAME', file 'YOUR_DATABASE_NAME_log' on file 1.
RESTORE LOG successfully processed 3 pages in 0.018 seconds (1.247 MB/sec).

Step:-4

i m assuming Transaction_Logs_3.trn is the file which we want to get the database data till yesterday,

one more file Transaction_Logs_4.trn is there but our requirement is till file Transaction_Logs_3.trn only

RESTORE LOG YOUR_DATABASE_NAME
FROM DISK = 'D:\YOUR_DATABASE_PATH\Transaction_Logs_3.trn' WITH RECOVERY

The above command will results in following output

Processed 0 pages for database 'YOUR_DATABASE_NAME', file 'YOUR_DATABASE_NAME' on file 1.
Processed 2 pages for database 'YOUR_DATABASE_NAME', file 'YOUR_DATABASE_NAME_log' on file 1.

RESTORE LOG successfully processed 2 pages in 0.064 seconds (0.190 MB/sec).

after the successfull operations of the above step the database is now ready for use,

till the time it is in RECOVERYMODE the same will not be available for use.

In the example above, we restore the database to the end of the 2nd last transaction log.

If we want to recover our database to a specific point in time before the end of that transaction log,
then we must use the STOPAT option.

The script below restores the fourth transaction logs in the log sequence to 3:00 AM - time just before the database gets currupted.

RESTORE LOG YOUR_DATABASE_NAME
FROM DISK = 'D:\YOUR_DATABASE_PATH\Transaction_Logs_4.trn'
WITH STOPAT = N'8/30/2010 3:00:00 AM', RECOVERY

MS SQL DATABASE/LOGS Shrinking ( Tested with MsSQL - 2005, 2008 )

Reference By Mr.Varun Dhavan (Database Expert Microsoft)

Q:- What does database shrinking means ?
Ans:- In a SQL Server database, each file within a database can be reduced to remove unused pages.
Although the Database Engine will reuse space effectively, there are times when a file no longer needs to be as
large as it once was; shrinking the file may then become necessary. Both data and transaction log files can be reduced, or shrunk.
The database files can be shrunk manually, either as a group or individually, or the database can be set to shrink automatically at specified intervals.


Q:-What are best practices and implications of shrinking ?
Ans:- Consider the following information when you plan to shrink a database or file:

1) A shrink operation is most effective after an operation that creates lots of unused space, such as a truncate table or a drop table operation.
2) Most databases require some free space for regular day-to-day operations. If you shrink a database repeatedly and notice that the database size grows again, this indicates that the space that was shrunk is required for regular operations. In these cases, repeatedly shrinking the database is a wasted operation.
3) A shrink operation does not preserve the fragmentation state of indexes in the database, and generally increases fragmentation to a degree. For example, you should not shrink a database or data file after rebuilding indexes. This is another reason not to repeatedly shrink the database.
4) Unless you have a specific requirement, do not set the AUTO_SHRINK database option to ON.


Q:- How does shrinking of a log file happen ?
Ans:- A log file is shrunk when you issue the following command to the SQL Server:
DBCC SHRINKFILE ('logical file name', targetsize)


Q:- How do I know if a log file can be shrunk ?
Ans:- To understand whether the log file can be shrunk, you will need to fire the following commands and understand their outputs.

The first command that needs to be fired is:
DBCC SQLPERF(logspace)
This will let us know what percentage of the log file is actually in use. The lower the percentage, the more the file can be shrunk.


SHRINKING DATABASE T-LOG FILES ROOT-CAUSE

Step we followed to shrink the T-log file of the database ?

Step 1. Back up the transaction log file to make most of the active virtual log files inactive.
Therefore, the inactive virtual log files can be removed in a later step.
To do this, run a Transact-SQL statement that is similar to the following Transact-SQL statement.

BACKUP LOG TO DISK = ''

Step 2. Shrink the transaction log file. To do this, run a Transact-SQL statement that is similar to the following Transact-SQL statement.
DBCC SHRINKFILE (, ) WITH NO_INFOMSGS


Root cause: Why the Transaction-log files grown so huge ?

1. Databases while running in FULL recovery model and When the transaction logs grow to an unacceptable limit,
you must immediately back up your transaction log file. While the backup of your transaction log files is created,
SQL Server automatically truncates the inactive part of the transaction log.
The inactive part of the transaction log file contains the completed transactions, and therefore, the transaction log file is no
longer used by SQL Server during the recovery process. SQL Server reuses this truncated,
inactive space in the transaction log instead of permitting the transaction log to continue to grow and to use more space.

2. If Replication on any databases is in active mode and SQL Server Agent is down then due to this, huge pile of transaction that
were pending to replicated, however could not be replicated as the Replication Agent jobs were not running

Wednesday, December 1, 2010

Enable Snort Syslog format and Redirect it to any required server

to enable Snort Syslog and Redirect it to any required server you need to modify the snort.conf file

output alert_syslog: host=ServerIP:514, LOG_AUTH LOG_ALERT LOG_INFO LOG_DEBUG

Overriding the default Servlet in Tomcats WEB-INF/web.xml

To Override the Global Tomcat Settings use the below Code for DIR Listings
<servlet>
        <servlet-name>DefaultNoListing</servlet-name>
        <servlet-class>org.apache.catalina.servlets.DefaultServlet</servlet-class>
        <init-param>
            <param-name>debug</param-name>
            <param-value>0</param-value>
        </init-param>
        <init-param>
            <param-name>listings</param-name>
            <param-value>true</param-value>
        </init-param>
        <load-on-startup>1</load-on-startup>
</servlet>
<servlet-mapping>
<servlet-name>DefaultNoListing</servlet-name>
<url-pattern>/</url-pattern>
</servlet-mapping>

Configuring tomcat to authenticate using windows Active Directory

<Realm className="org.apache.catalina.realm.JNDIRealm" debug="99"

           connectionURL="ldap://youradsserver:389/"
           alternateURL="ldap://youradsserver:389/"         
           userRoleName="member"
           userBase="cn=Users,dc=yourdomain"
           userPattern="cn={0},cn=Users,dc=yourdomain"
           roleBase="cn=Users,dc=yourdomain"
           roleName="cn"
           roleSearch="(member={0})"
           roleSubtree="false"
           userSubtree="true" 
   />
and define the role in the tomcat-users.xml and the web.xml of your application
edit webapp_root/WEB_INF/Web.xml file as follows:
<security-constraint>
   <display-name>your web app display name</display-name>
   <web-resource-collection>
     <web-resource-name>Protected Area</web-resource-name>
     <url-pattern>*.jsp</url-pattern>
     <url-pattern>*.html</url-pattern>
     <url-pattern>*.xml</url-pattern>
   </web-resource-collection>
   <auth-constraint>
     <role-name>yourrolname(ADS Group)</role-name>
   </auth-constraint>
 </security-constraint>
 <login-config>
   <auth-method>FORM</auth-method>
   <form-login-config>
     <form-login-page>/login.jsp</form-login-page>
     <form-error-page>/error.jsp</form-error-page>
   </form-login-config>
 </login-config>
 <security-role>
   <description>your role description</description>
   <role-name>yourrolename(i.e ADS group)</role-name>
 </security-role>

Tomcat Clustering

Tomcat Clustering

IHS with WebSphere Application Server

IHS with WebSphere Application Server

IIS with WebSphere Application Server

Once the WebSphere Application Server 4.0 is installed successfully then it can be configured with the IIS 6.0 server by creating plugins for the same.
Load a plugin iisWASPlugin.dll

once the plugin/filter is created check its priority if it is being loaded properly then it will be with High Priority as shown below.

The next step is to create a virtual directory sePlugins with the scripts & executables permission for the created filter iisWASPlugin.

Next final step is to restart the www services and the same can be checked by using the default application's url like http://www.someserver.com/very_simple.jsp the very_simple.jsp will be served by the websphere application server which will servers the request at the port 80.

Apache Server with Tomcat server

The simplest configuration is described. It assumes you already have Tomcat 5.5 and Apache 2.0 (instructions for Apache 1.3 is also provided) installed and running.

The instructions are applicable (have been tested) for Windows as well as Linux platform.

Assume you want to map test directory of Apache to the mytest web application of Tomcat. Change the name appropriately to suit your configuration.

1. Shutdown Apache & Tomcat Server
2. Add the following lines to httpd.conf (in conf directory of Apache base directory)

LoadModule proxy_module modules/mod_proxy.so
LoadModule proxy_http_module modules/mod_proxy_http.so
ProxyPass /test/ http://localhost:8081/mytest/
ProxyPassReverse /test/ http://localhost:8081/mytest/

Note 1: Replace localhost with the appropriate IP address or hostname of the server where Tomcat is installed.

Note 2: On older Apache 1.3 you will have to use libproxy.so instead:
LoadModule proxy_module modules/libproxy.so
AddModule mod_proxy.c

IIS with Apache Tomcat Server

IIS with Apache Tomcat Server

Integration of Web Servers with Application Servers

IIS with Apache Tomcat Server
Apache Server with Tomcat server
IIS with WebSphere Application Server
IHS with WebSphere Application Server

Tomcat 6 - Discloses username="tomcat" password="s3cret" roles="manager"

Tomcat 6 - Discloses username="tomcat" password="s3cret" roles="manager"

Custom Error page configurations not working with Tomcat 6.x and while using the Manager application of tomcat if user tries the invalid username / passwords it discloses the 401 unauthorized page as shown below.

to manage the issue either disable the Manager application or modify the error page so that  username="tomcat" password="s3cret" roles="manager" this string can be avoided and for the same comment out the below code of the 401.jsp file located at D:\ApacheSoftwareFoundation\Tomcat6.0\webapps\manager and restart the tomcat and you are done.

<pre>
&lt;role rolename="manager"/&gt;
&lt;user username="tomcat" password="s3cret" roles="manager"/&gt;
</pre>

Tomcat Startup failure on Win2k8 R2

Getting error while Starting tomcat server 5 on windows 2008 R2 - 64bit operating system as the packages used were supported for 32 bit versions.







[402  prunsrv.c] [error]
The system cannot find the file specified

[1246 prunsrv.c] [error]
Load configuration failed

i have resolved this error simply by installing the JRE 6 and Tomcat 6 for 64 bit windows system and it worked successfully and also it installed this as Windows service which is common isssue if you are installing 32 bit package.

Packages used are
jre-6u4-windows-x64.exe
apache-tomcat-6.0.29.exe


Download links for 64 bit packages - ( JRE )
http://www.start64.com/index.php?Itemid=114&id=1792&option=com_content&task=view

Tuesday, November 30, 2010

Microsoft Web Server - IIS 7.0 Hardening Recomendations

Important checks which are very crucial @ IIS Web Server

All Unknown CGI Extentions must be -> Prohibited
All Unknown ISAPI Extentions must be -> Prohibited
Active Server Pages must be subject to your requirement -> Prohibited
Internet Data Connector must be -> Prohibited
Server Side Includes must be -> Prohibited
WebDAV must be -> Prohibited
Server Header Info must be -> DISABLED
Application Server Running with its own UserAccount must be -> ENABLED
Error Disclosures must be -> DISABLED
Server OS informations/Physical Path Disclosures must be -> DISABLED

Really Great and very useful Post by Mr.Steve Schofield
Reference :- http://forums.iis.net/t/1127617.aspx

1) Run as applicationpoolidentity
2) Uninstall any modules that aren't used, especially authentication modules.  Not having modules reduces the surface attack. 
 If you install additional modules, run at website level, don't load at server level,
 use the web.config to load the modules in the <system.Webserver> section.
3) Look at using Request Filtering or urlscan to block sql injections
4) You can use host-headers to help reduce automated ip-based bot attacks.
5) Do not enable remote management, it's disabled by default
6) Don't install FTP, SMTP services.
7) Run each website in their own application pool
8) Lockdown any delegated permissions or remove them all together. OS, App suggestions
9) Run Security Config wizard this does OS level changes.   Definitely spin up a test VM or box to test SCW before applying at GPO level
http://weblogs.asp.net/steveschofield/archive/2008/10/26/how-to-use-security-configuration-wizard-in-windows-server-2008.aspx
10) Run Windows firewall, block all but 3389, 80, 443, echo reply (for monitoring and pings). 
11) Place Data on a separate drive, remove default NTFS permissions,
12) Keep up on security patches, service packs.
13) Run asp.net apps in medium or partial trust if possible.  Don't install DLL's in the GAC (global assembly cache)
14) Enable auditing in the local security policy (or GPO).
15) Run Anti-virus software.
16) Enable custom errors errors so unhandled errors aren't displayed remotely
17) Most web applications need to be properly tested for hacking, unhandled exceptions, etc..  IIS 7 itself is solid, the applications need to be both load tested and how they handle such situations.
18) Run 64 bit version of W2K8 or R2.

Simple steps to generate the Certificate for Apache Server 2.2 using openssl

1) Go to the command prompt
2) cd C:\Program Files\Apache Software Foundation\Apache2.2\bin
3) set OPENSSL_CONF=C:\Program Files\Apache Software Foundation\Apache2.2\conf\openssl.cnf ( if Required )
4) openssl
5) genrsa -des3 -out test.com.key 1024
6) rsa -in test.com.key -out server.pem
7) req -new -key test.com.key -out test.com.csr
8) Once the above steps completed total 3 files will be generated to the location C:\Program Files\Apache Software Foundation\Apache2.2\bin
            test.com.key
            server.pem
            test.com.csr
8) Get the Certificate from Certified Authority using csr file "test.com.csr"
9) Installing the Certificate (test.com.cer) Received from the Certified Authority
i) Copy the test.com.cer and server.pem into the "C:\Program Files\Apache Software Foundation\Apache2.2\conf"
ii) Now Edit the httpd.conf Uncomment Both the lines #Loadmodule ssl_module modules/mod_ssl.so and #Include conf/extra/httpd-ssl.conf
iii) Now Edit the httpd-ssl.conf file from location "C:\Program Files\Apache Software Foundation\Apache2.2\conf\extra" Change the from  SSLCertificateFile "C:/Program Files/Apache Software Foundation/Apache2.2/conf/server.crt" to SSLCertificateFile "C:/Program Files/Apache Software Foundation/Apache2.2/conf/test.com.cer"
10) Restart the Apache Server and Check the same.

Run your Tomcat server on HTTPS

Simple Steps to make your TOMCAT Run on HTTPS using KEYTOOL utility

1). keytool -genkey -alias www.mytest.com -keyalg RSA -keysize 2048 -keystore www_mytest_com.jks

2). keytool -certreq -alias www.mytest.com -file www_mytest_com.csr -keystore www_mytest_com.jks

3). keytool -import -trustcacerts -alias www.mytest.com -file D:\www_mytest_com.p7b -keystore www_mytest_com.jks
This will install Certificate and the Root Certificate associated with the same., sometimes this will not work and in that case try the below steps.

3.1). keytool -import -alias www.mytest.com -file D:\www_mytest_com.cer -keystore www_mytest_com.jks
3.2). keytool -import -trustcacerts -file D:\www_mytest_com_root.cer -keystore www_mytest_com.jks

4). Modify server.xml and Restart TOMCAT SERVER


**Note: By default Tomcat will look for your Keystore with the file name .keystore in the home directory with the default password changeit. The home directory is generally /home/user_name/ on Unix and Linux systems, and C:\Documents and Settings\user_name\ on Microsoft Windows systems

Apache 2 to Tomcat Connector Using Proxy

The simplest configuration is described. It assumes you already have Tomcat 5.5 and Apache 2.0 (instructions for Apache 1.3 is also provided) installed and running.

The instructions are applicable (have been tested) for Windows as well as Linux platform.

Assume you want to map test directory of Apache to the mytest web application of Tomcat. Change the name appropriately to suit your configuration.

1. Shutdown Apache & Tomcat Server
2. Add the following lines to httpd.conf (in conf directory of Apache base directory)

LoadModule proxy_module modules/mod_proxy.so
LoadModule proxy_http_module modules/mod_proxy_http.so
ProxyPass /test/ http://localhost:8081/mytest/
ProxyPassReverse /test/ http://localhost:8081/mytest/

Note 1: Replace localhost with the appropriate IP address or hostname of the server where Tomcat is installed.

Note 2: On older Apache 1.3 you will have to use libproxy.so instead:
LoadModule proxy_module modules/libproxy.so
AddModule mod_proxy.c

Tomcat Hardening Recomendations

1. use an unprivileged user account to run the  server.
2.use a firewall before your server
3. Disable the connectors you dont need  in server.xml
4. disable the tomcat's admin/manager web application completely or configure it that way that it needs proper username/passwort and connection from well known hosts
5. Disable the examples application
6. use apache http server to forward the request to the tomcat server.
7. bind tomcat to those IPs and ports only which you need, don't bind to any
8.Use server-minimal.xml instead of server.xml (make security life simpler;-)
9. check what you allow in tomcat's default context.xml, web.xml and anything below your configured host 10. use a special user to run tomcat, don't use administrator/root for that
11. allow only that user to read all your files, disallow any other users
12. make all files read-only (except those tomcat needs to write to)

To Allow/Disallow access from Specifc port use following Valves.
<Valve className="org.apache.catalina.valves.RemoteHostValve" allow="10.6.1.*" deny="10.6.1.1"/>

By Defining Address Tag we can Allow the AJP Access from specific IP only as shown below
<Connector address="127.0.0.1" port="8009"   enableLookups="false" redirectPort="8443" protocol="AJP/1.3"  allowTrace="false" xpoweredBy="true"/>

<Valve className="org.apache.catalina.valves.AccessLogValve" directory="logs"  prefix="localhost_access_log." suffix=".txt" pattern="common" resolveHosts="false"/>
<Valve className="org.apache.catalina.valves.RemoteHostValve" allow="127.0.0.1,10.6.10.*" deny="10.6.10.2"/>

while Config of ADMIN/ Manager Application For tomcat Administration put admin.xml and manager.xml from the server's server\webapps location to the D:\ApacheSoftwareFoundation\Tomcat5.5\conf\Catalina\localhost and Restart the Tomcat Server to get the Changes
whenever making any change to the admin.xml again put the same in the above mentioned location and Restart

Always Allow ADMIN/Manager Application from Local/Intranet IP's don’t let them over Internet due to security Threat

For more informations on securing TOMCAT refere
http://www.owasp.org/index.php/Securing_tomcat
http://www.unidata.ucar.edu/Projects/THREDDS/tech/reference/TomcatSecurity.html