Finding Memory Leaks in Java Applications

Trying this out to analyze memory usage and find leaks in Java application:

http://blog.emptyway.com/2007/04/02/finding-memory-leaks-in-java-apps/

Pretty slick...

In Eclipse for my debug configuration I put this in the vm arguments box:

-Dcom.sun.management.jmxremote
-Dcom.sun.management.jmxremote.port=9000
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.ssl=false
-agentlib:hprof=heap=dump,file=/tmp/hprof.bin,
format=b,depth=10

A few more details for those not so familiar with Java:

jconsole.exe is found in the /bin directory in the directory where you installed the Java JDK. It is an application with a GUI so just double click on it to open it up. You then select your application from the list to view details about it.

jps.exe is run from command prompt. It is also found in the /bin directory in the directory where you installed the Java JDK. Go to a command prompt (start menu, run, enter "cmd" hit enter) and navigate to that bin folder or alternatively you can type out the full path to jps.exe to run it.

jmap.exe is also in the /bin directory. You also would run that from the command line by typing in the location where you want to write the file and the process id as described in the above article.

jmap -dump:format=b,file=/tmp/java_app-heap.bin 15976

Once that file is created,run jhat as directed.

jhat -J-Xmx326m /tmp/java_app-heap.bin

Then in browser go to:

http://localhost:7000

You can browse that data to find which objects are most in use, how much memory they are taking up, etc.

If a very slow leak can run the app in production using the above to output file over time and then view it.

Seems like support for Eclipse Memory Analyzer and Eclipse TPTP are dwindling but this method above is pretty simple.

Postini, IMail, SSL/TLS

Here's a run down on setting up Postini with IMail. Really quick...disclaimer in case of error but here are my notes...

Postini

You probably all run some kind of mail software with spam blocking capabilities. However the painful thing is that spammers are out there inserting mail directly into your mail server and overloading spam boxes all the time. This takes up resources on your mail server(s) and clients.

What's kind of neat about Postini is that it's a filtering service in front of your mail server. You can force all inbound and outbound mail to go through Postini if you have your own mail server or a service that offers Postini. By setting up Postini correctly a lot of the garbage bombarding your server can be stopped before it ever gets that far.

Postini was bought out by Google and as you can imagine has a tad bit more mail to analyze than the average company hosting an email system. By passing all the garbage through them first, I would hope they can do some pretty good analytics on mail going through their system and figure out how to weed out the junk, the same way they figure out how to put the best web sites at the top of the search engine results [hopefully!]

Postini blocks viruses, spam and has a bunch of cool features like allowing you to block IP ranges from sending mail to your server. Since a lot of spam comes from certain parts of the world, if you don't do business in those areas you can simply block out entire spam friendly networks completely. I found that blocking out everything but ARIN significantly reduced my spam on a particularly old domain that is probably in every spammer's list in existence.

The other cool thing about Postini is that it allows you to force TLS between two parties - for regulatory copmliance or simply because you want to know your email transmission is encrypted end to end (which can be hard to do).

These are just a few of the highlights of Postini service.

The biggest problem with Postini is that, if you're a small business, getting support is challenging. You don't always get answers and when you do they are vague or not actually answering the question you asked or simply don't resolve the problem. Lately, however, I haven't had a ton of questions which is good. The service has been pretty reliable.

IMail

If you're not interested in spending mega-bucks on Exchange, IMail is a simpler, less expensive option and the company provides very good support. When it comes to email that's important to me because I find email to be a royal pain. As noted in a recent episode of American Greed, spam is the backbone of organized Internet crime. They are all trying to get into your in box to get you to click a link - or something.

When people's email isn't working - they want it fixed. Now. Although I'm pretty handy with Java, open source typically comes with no or minimal support. Additionally open source has so many people contributing that are not quite held accountable in the same way I would expect a company with paid staff to be. Mostly, I'd rather pay a bit more and get someone who can help me fix things fast than futz around with user groups and mailing lists or reverse engineering code to find a fix to a type of technology I find completely annoying because it is hard to secure, unreliable and always under attack. I prefer to deal with email as little as possible. I just want to call someone and get an answer. There is one person at Ipswitch in particular I have talked to over the years who is extremely knowledgeable about IMail and was able to quickly answer all my questions during set up.

For me running on IIS was kind of a pain because I prefer Java and it's another thing to manage, but I've used Imail in the past and it's reliable with good support as mentioned. For those who like and know IIS this would be a plus. Setting up the SSL cert was much easier with IIS than Java. The web site runs a little slow but going to look into how that can be tweaked. Using IMail is, for the most part, a piece of cake. The only thing I found challenging (having a basic understanding of mail servers) was getting the SSL cert installed and finding all the parts and pieces to make it work with Postini.

Set Up

--IMAIL--
> Purchase or get trial at IPSwitch.com
> Install - call for help as needed.
> Note that MX records will be pointed at Postini (see below) not your mail server.

--POSTINI--
> Set up an account and login to the System Administrator
> Click on the Orgs & Users tab
> Select your top level account from the drop down list of orgs
> Click the link (name) of your account (top node in the list)
> Enter a new org name in the box at the top and click the "add" button
> Click on orgs & users tab to get the full list of orgs
> Click on the name of the org you just created to view the settings.
> Click on the "general settings" link 3/4 down the page
> Change "email config type org" to Yes and save [allows mail server settings for this org]
> Click on "DNS Instructions" for MX records to use...

--EDIT YOUR DNS--
> edit your MX records to include the specified servers (note that each domain should have the domainname itself in the MX record - replace the part that says yourdomain with the domain for which you are setting up email)

--POSTINI--
> Click on "domains" under the tabs and add a domain
> Click on "users" under the tabs and createa user
> Click on "Inbound Servers" tab, "Delivery Manager", then "Edit" to add your mail servers. If you have your own mail server choose "Use my own mail server" and % to balance load to each server.
> There are other cool features you can look into later...manually block mail from certain IPs (bottom right when you click on "Inbound Servers"), force TLS between certain domains, etc.

--IMAIL--
> In your mail server allow relay from Postini (ONLY so you don't create an open relay for spammers!) In the latest version of Imail you would click on "Services" on the left, then double click on "Imail SMTP service".
> Under "General" click on the "Addresses" tab. Select "Group" from the drop down and enter the Postini IP addresses specified for your outbound mail configuration. The IP addresses depend on which server number you were assigned when you signed up for the service. The server number is in your MX records specified in the DNS instructions. More in this document:
Postini Outbound Services

--POSTINI--
> Click on "Outbound Servers" tab, and at the bottom left choose "Add Another Outbound Email Server" and add your mail server(s) that will be used for sending bounces back to your users.

--IMAIL --
> Set up webmail per the IMail instructions.
> If you want webmail to show up on it's own domain with no directory at the end and no access to the administrator and other things from that domain, set up a second web site on IIS and point it specifically at the webmail folder instead of the parent directory. In other words instead of mail.mydomain.com/web_mail_folder to get to mail you point IIS web site at the web_mail_folder and then can get to that same thing by going to mail.mydomain.com (if that doesn't make sense...call IPSwitch. They are very helpful.)

-- SSL CERTIFICATE (webmail and mail traffic) --
> In IIS create a cert request. I used Digicert.com and they were extremely easy to work with. Click on the IIS link on the left side of their home page and scroll to the bottom to get instructions for IIS cert request.
> When you get back the certificate, install it in IIS (for the domain you're running webmail on).
> Test your webmail domain and make sure it's accessible via SSL (https://mail.mydomain.com)
> For Digicert there were some chained certificates I needed to install in IMail. Originally exporting from IIS didn't work because the output didn't include the full chain. So...
> Use the MMC to export the cert with the full chain of certs:
http://www.digicert.com/ssl-support/pfx-import-export-iis-7.htm
> These "unsupported" instructions will get you two files that have the private key and the cert in separate files - I don't know if there's any issue with this...but I did it and it worked.
http://kb.imailserver.com/cgi-bin/imail.cfg/php/enduser/std_adp.php?p_faqid=42&p_created=1218038946
> You'll need the full chain in your cert file as explained in this article:
http://support.ipswitch.com/kb/IM-20010425-DM01.htm

-- PORTS AND FIREWALL--
> You'll want to make sure spammers can't inject mail straight into your mail server, so set up your firewall to only allow SMTP to Postini
> In order to let users connect, you set up a separate port for them to connect to and force authentication on that port. In Imail there's an option to do this on the "advanced" link for SMTP server settings.
> I also tried to use that same port for SSL hoping all traffic would be SSL encrypted on that alternate port but found the following info at the bottom of this link:

http://support.ipswitch.com/kb/IM-20050428-DM01.htm

Note for Outlook and Outlook Express users: If you tell your client to use SSL over SMTP and set the port to anything other than 25 the client will attempt to negotiate the SSL connection before sending the EHLO as if it was communicating over a dedicated SSL port. This will not work on the extra port. Users must issue EHLO and then STARTTLS to use SSL over the alternate SMTP port.

When users tried to enter the port and check "SSL" in outlook didn't work. They had to uncheck it to get this to work. However another mail connection I used from an alternate piece of software didn't work. By moving SSL to a separate port I believe that Outlook will attempt to use TLS if available. What I wonder is - what if something happens and it's not? The problem here is that with Postini you can force TLS between your mail server and Postini and between Postini and other mail servers, but for people connecting from their machine to our mail server - we also want to make sure that portion of the transmission is encrypted. Need to set up some alternate type of monitoring to enforce this potentially.

> So for now users are set up to connect to alternate port with SSL unchecked.
> Set up SSL port for POP/IMAP also and provided that port with "server requires SSL" setting.

TLS is cool because it runs on the same port as SMTP so takes up less ports, easier for users to configure and uses TLS when available otherwise not - but for connections from users to the mail server, wondering how that can be enforced and monitored as noted.

-- DICTIONARY ATTACKS --

Shortly after set up we were plagued by "dictionary attacks" (see my twitter account for more on that http://twitter.com/teriradichel

To prevent or at least slow down dictionary attacks - where someone just hits your web server with every possible valid email they can think of in rapid succession until they find a good one - edit SMTP service settings in the box that says "Dictionary" attacks and limit the number of attacks per session, failures, etc.

-- TRANSFERING MAIL TO MAIL SERVER --

If you're transferring users from another mail server you might also be interested in:

How to transfer your users and mail from your old mail server to new:
http://kb.imailserver.com/cgi-bin/imail.cfg/php/enduser/std_adp.php?p_faqid=716&p_created=1247145176&p_sid=gOAbiCfk&p_accessibility=0&p_redirect=&p_lva=&p_sp=cF9zcmNoPTEmcF9zb3J0X2J5PWRmbHQ6MSZwX2dyaWRzb3J0PSZwX3Jvd19jbnQ9MTU4LDE1OCZwX3Byb2RzPTAmcF9jYXRzPTAmcF9wdj0mcF9jdj0mcF9zZWFyY2hfdHlwZT1hbnN3ZXJzLnNlYXJjaF9ubCZwX3BhZ2U9MiZwX3NlYXJjaF90ZXh0PW1vdmUgbWFpbA**&p_li=&p_topview=1

Basically to move mail you transfer all your users over as noted above. Then you can connect via IMAP in outlook to both mail servers, and copy all the mail from the inbox of one server to the inbox of the other. There may be other folders to copy as well depending on what the user has set up.

Probably missing a few things here but that's the gist of it.

Skip Tests Running Maven Install From Command Line

mvn -Dmaven.test.skip=true install

JDK version - Eclipse, Maven

Do you know what compiler Maven is really compiling with when you choose "install"?

I am able to get a project in Maven-Eclipse-Plugin to compile with a specified compiler when using "Maven build..." option. But when I choose "Maven install" it has a mind of its own and decides to use an old compiler version. This may cause a project to fail to build due to incompatibilities (generics) or introduce security issues because various security flaws have been fixed in newer versions of Java.

What really bugs me is that this happened without me changing anything as noted in this previous post, and is kind of frustrating because have to spend time futzing with this instead of getting work done:

Maven-Eclipse compiler issue

Didn't get so lucky with some simple tweaking in my last post, so added compiler variable to a profile in settings.xml like this:

Specify Maven Compiler JDK version in settings

I added a new profile to my Maven settings.xml file as specified in link above. I called my compiler variable MY_JAVA_HOME and pointed it to the JDK I want to use.

Then I added the maven compiler plugin to my project by right clicking on the project node in Eclipse, choosing Maven (I'm using the Maven-eclipse-plugin as specified in other posts which is the source of the Maven right click menu in Eclipse), then adding this plugin: maven-compiler-plugin.

I then replaced configuration node as specified above but using the variable I created in my settings.xml profile:

        <configuration>
          <verbose>true</verbose>
          <fork>true</fork>
          <executable>${MY_JAVA_HOME}/bin/javac</executable>
          <compilerVersion>1.6</compilerVersion>
        </configuration>

Seems to work, however there is a warning in the above link that this does not necessarily force the compiler to use the specified JDK version (which I find kind of odd also - use my configuration and throw an error if it won't work).

I actually dropped out of Eclipse and ran mvn install from the command line to get a bit more control over what was going on and was able to get all my projects to compile and install again.

I personally don't like when my software applications do things I don't ask them to do and don't tell me about it - and it's not the way I have it configured.

Securing Encryption Keys

I see a lot of articles about encryption but not a lot of information about protecting encryption keys. What good does encryption do if someone has the key? That's like locking your house when you leave for work but leaving the key hanging on the door.

Let's say you encrypt a configuration file. Now you start up your app and you need to decrypt that file. In order to decrypt there's a key somewhere most likely that allows decryption of that file. Where is that key stored? Do you have to type it in when you run the application? Do you have to look it up in a database? How do you secure access to that database? Do you put it in another file? How do you secure access to that file over and above the way you secured access to the configuration file itself?

Whoever has access to that key can decrypt your data - do network admins have access to the file with the key? Do database administrators have access to the key stored in the database? Can a hacker access the key through a backdoor or SQL injection under the permisissions of the user account that runs your web application? Is the key sent unprotected over the network or with weak SSL encryption? Is the key floating in memory on your machine in easy to spot format? Is it embedded in a Java class that is simple to decompile using tools on the web? Are your files and databases backed up to a back up device accessible by third parties? How do you implement checks and balances required for PCI Compliance in terms of encryption key storage and retrieval?

Here is some information I was able to find on the topic:

Some suggestions for protecting encryption keys
http://it.toolbox.com/wiki/index.php/Protect_encryption_keys_in_memory,_files,_databases,_and_during_transport_between_system_processes

PCI Encryption suggestions:
http://forums.sun.com/thread.jspa?threadID=5433515

Limited information but discussing the issues around keys:
http://www.symantec.com/connect/articles/introduction-encryption

Encryption key policy from SANS:
http://www.sans.org/security-resources/policies/User_Encryption_Key_Protection_Policy.pdf

Intel loses DRM Protection Key 9/17/2010 - key management is not simple...
http://www.voltage.com/products/data_protection.htm

Voltage has been in the encryption / key management space for a while - but how are your keys secure when given to a third party?
http://www.voltage.com/products/data_protection.htm

Self Signed Certs in IIS 7

Nice job of expaining how to set up a self-signed cert in IIS 7.

Self-Signed certs in IIS 7

Maven Mysteries - Generics and Compiler Issues

Maven by default compiles code with a lower compiler version than 1.5 which has support for generics.

Today I was programming along when suddenly a project that has been compiling for a while suddenly stopped compiling (using Maven + Eclipse) and Maven started griping about the project code's inclusion of generics when this is not supported prior to JDK 1.5. Hmm. I had this working before so why did it suddenly start failing? Who knows.

So I go in and reset the project to use the 1.6 compiler by right clicking on the project node in Eclipse, choosing compiler and changing the settings.

No dice.

I then right clicked on the project node, chose Maven (I'm using Maven plugin as described in another post) and choose "Update Project Configuration". Right click on the pom, choose Run As and Maven Install...well then I started getting compiler errors in Eclipse I wasn't getting before. I went to one of the errors, right clicked and told it to change the project to use JDK 1.5 which supports generics.

Ok now it compiles...but now the setting changes I just made say 1.5 instead of 1.6. Why didn't it work when I set the project to use JDK 1.6? So I go back to the project node, change the compiler setting to JDK 1.6. Still compiles. That's good.

Then I go right click and choose Maven and Update Project Configuration. Running Maven Install still works.

OK great. Whatever.

Classloaders In Depth

Digging into the details of classloads has been on my list of research items for a while. I understand the concepts and the implications - what if a class gets loaded other than the one you intend to be running in your app and you're not really running the code you think you are?

Anyway...ran into this error today...

java.lang.LinkageError: loader constraint violation: when resolving method "[class]" the class loader (instance of [classloader1]) of the current class, [class], and the class loader (instance of [classloader2]) for resolved class, [class], have different Class objects for the type [class in conflict] used in the signature

Which led me to this great blog post which does a great job of epxlaining the nitty gritty of class loaders and potential issues with self-first and parent-first class loading...

Java Class Loaders

Nice.

Oh and by the way, I fixed this by changing the package of the class in my web app so it was different than the package of the conflicting class loaded by the web server. This may not be the right solution in all cases - you may want to remove the class from your web app and use the parent class or vice versa instead.

Strange Network / Web Site behavior

Something very odd happened when I transferred web sites from one network to another, and view them using one particular network provider.

If I hit the web sites in rapid succession the images go all wacky and some images will get displayed multiple times when they are only supposed to be displayed once, for instance. These are straight html pages - about as basic as you can get.

I cannot reproduce this skewing of the web pages when working locally on the machine that is hosting the web site.

I also noticed that when I strolled down to the local coffee shop with my laptop I could not reproduce this funky behavior. This leads me to believe that the problem is due to the network or software thereon of the ISP at the particular location having the problem. It seems like something is capturing and altering the results of the request in transit and at some point before reaching my computer they are getting skewed.

Will have to research further.

7/15/2010

Interesting...last night I uncovered that my mail seemed to be going through a proxy server. They claimed that my network traffic was not going out through any proxy servers previously, but I asked them to double check the web traffic since obviously something was affecting the email traffic and sending it where it shouldn't be going.

Magically today, I have not yet been able to reproduce whatever was causing web sites to be skewed. I have not yet tested on Comcast network again.

Database restore script - roles, users, permissions and schemas

I noted some code for creating some database roles and assigning users to them the other day. Here's a more extensive script that can be used when transfering databases between environments (say dev, test and production) where you want to assign users to specific schemas and roles, clear out permissions, and/or restore database users after restoring a database. Using a script is preferable to doing everything by hand because it can produce more consistent results and will be faster. Additionally you can use the same script in different environments and set up different users and/or schemas as needed.

This script should save a lot of time, though could be even more automated (creating logins for instance).

Additionally if you have existing users with appropriate permissions you may want to (I have recently learned) assign the new user in the newly restored database the old sysid to keep those permissions in tact.

Another item of note - all permissions and roles are stored in the master database. If you want to return to default roles and permissions, hopefully you have backed up the database and can restore the master database to retrieve those settings. Restoring the master database requires putting the SQL Server database in single user mode and there are a few tricks to that which I have around here somewhere...



/*

Before running this script:

a. create the user + pwd on the server

b. replace this everywhere found below with your new user name and schema:

set @newuser = 'test'
set @dbschema = 'dbs'

c. run script below which does the following:

add schema to all dbs
add user to all dbs
add all objects to schema @dbschema
add roles to dbs
add user to roles


*/



/*
STEP 1: add schema to every db
*/

Declare @dbschema varchar(10)
Declare @sql varchar(4000)

set @dbschema = 'dbs'

set @sql = 'IF ''?'' NOT IN (''master'', ''model'', ''msdb'', ''tempdb'') ' +
' BEGIN ' +
' use ? ' +
' declare @s varchar(255) ' +
' if not exists(select name from sys.schemas where name=''' + @dbschema + ''')' +
' begin ' +
' set @s = ''create schema ' + @dbschema +
' '' exec (@s) ' +
' end ' +
' end '

print @sql
exec master.dbo.sp_MSforeachdb @sql

GO

/************
Step 2: change schema of all user defined objects
************/

Declare @sql varchar(2000)
Declare @cntitems int
Declare @dbschema varchar(20)
Declare @oldschema varchar(20)
Declare @itemname varchar(128)
Declare @db varchar(128)

set @dbschema = 'dbs'

if (select object_id('tempdb..##objnames')) is not null
drop table ##objnames

create table ##objnames (itemname varchar(128), oldschema varchar(128), dbname varchar(128))

set @sql = 'IF ''?'' NOT IN (''master'', ''model'', ''msdb'', ''tempdb'') ' +
' BEGIN ' +
' use ? ' +
' insert into ##objnames select t.name, s.name, ''?'' from ' +
' sys.all_objects t ' +
' left outer join sys.schemas s ' +
' on s.schema_id = t.schema_id ' +
' where is_ms_shipped = 0 and parent_object_id = 0 ' +
' and type_desc <> ''system_table'' and ' +
' t.name not like ''dt_%'' and t.name not like ''%#'' and s.name <> ''' + @dbschema +
' '' END'

print @sql

exec master.dbo.sp_MSforeachdb @sql

/*****
loop through the objects we found and change schemas
******/

select @cntitems=count(*) from ##objnames

While @cntitems > 0
begin
select top 1 @itemname=itemname, @oldschema = oldschema, @db = dbname from ##objnames

delete from ##objnames where itemname = @itemname and oldschema=@oldschema and dbname=@db

if (@itemname like '%-%')
print ('ERROR: cannot update item with - in name: ' + @itemname)
else
begin
set @sql = 'use ' + @db + ' ALTER SCHEMA ' + @dbschema + ' TRANSFER ' + @oldschema + '.' + @itemname
print (@sql)
exec (@sql)
end
select @cntitems=count(*) from ##objnames

end


/*********
step 3: delete orphan users now that schemas are fixed
**********/

-- var for exec sql

-- vars are for db loop below
declare @CNT int
declare @name char(128)
declare @sid varbinary(85)
declare @c int
declare @hexnum char(100)

set nocount on


/*
You may need to change the database owner...

set @sql = 'select @cnt = count(*) from master..syslogins l right join ' +
rtrim(@db) + '..sysusers u on l.sid = u.sid' +
' where l.sid is null and u.name = ''DBO'''
exec sp_executesql @sql,N'@cnt int out',@cnt out

if DB is not mapped to login that exists map DBO to SA
if @cnt = 1
begin
--print 'exec ' + @db + '..sp_changedbowner ''SA'''
-- exec sp_changedbowner 'SA'
end -- if @cnt = 1
*/

Declare @sql varchar(1000)
Declare @db varchar(50)

if (select object_id('tempdb..##orphans')) is not null
drop table ##orphans

create table ##orphans (orphan varchar(128), dbname varchar(128))

set @sql= 'BEGIN ' +
' use ? ' +
' insert into ##orphans select u.name, ''?'' from master..syslogins l right join ' +
' sysusers u on l.sid = u.sid ' +
' where l.sid is null and issqlrole <> 1 and isapprole <> 1 ' +
' and (u.name <> ''INFORMATION_SCHEMA'' and u.name <> ''guest'' and u.name <> ''dbo'' ' +
' and u.name <> ''system_function_schema'' ' +
' and u.name <> ''sys'') and u.name <> ''MS_DataCollectorInternalUser'' ' +
' END '

exec master.dbo.sp_MSforeachdb @sql

select @cnt = count(*) from ##orphans

WHILE @cnt > 0
BEGIN

select top 1 @name= orphan, @db = dbname from ##orphans
delete from ##orphans where orphan = @name

set @sql = 'exec ' + rtrim(@db) + '..sp_revokedbaccess ''' + rtrim(@name) + ''''
print @sql
exec (@sql) -- drop the orphan from the db

select @cnt = count(*) from ##orphans

end
go

/**********
Step 4: create roles and assign user to roles
*********/
Declare @newuser varchar(20)
declare @dbschema varchar(20)
Declare @sql varchar(1000)

set @dbschema = 'dbs'
set @newuser = 'test'

/* create the cto roles on each database */

set @sql = 'IF ''?'' NOT IN (''master'', ''model'', ''msdb'', ''tempdb'') ' +
' BEGIN ' +
' use ? ' +
' IF NOT EXISTS (select * from sys.database_principals where name=''dbs_select'') ' +
' create role dbs_select ' +
' end '

print @sql

exec master.dbo.sp_MSforeachdb @sql

set @sql = 'IF ''?'' NOT IN (''master'', ''model'', ''msdb'', ''tempdb'') BEGIN ' +
+ ' use ? ' +
' IF NOT EXISTS (select * from sys.database_principals where name=''dbs_execute'') ' +
'create role dbs_execute ' +
' end '

print @sql

exec master.dbo.sp_MSforeachdb @sql

set @sql = 'IF ''?'' NOT IN (''master'', ''model'', ''msdb'', ''tempdb'') BEGIN ' +
+ ' use ? ' +
' IF NOT EXISTS (select * from sys.database_principals where name=''dbs_delete'') ' +
'create role dbs_delete ' +
' end '

print @sql

exec master.dbo.sp_MSforeachdb @sql

set @sql = 'IF ''?'' NOT IN (''master'', ''model'', ''msdb'', ''tempdb'') ' +
' BEGIN ' +
' use ? ' +
' IF NOT EXISTS (select * from sys.database_principals where name=''dbs_update'') ' +
' create role dbs_update ' +
' end '

print @sql

exec master.dbo.sp_MSforeachdb @sql

set @sql = 'IF ''?'' NOT IN (''master'', ''model'', ''msdb'', ''tempdb'') BEGIN ' +
+ ' use ? ' +
' IF NOT EXISTS (select * from sys.database_principals where name=''dbs_insert'') ' +
' create role dbs_insert ' +
' end '

print @sql

exec master.dbo.sp_MSforeachdb @sql


/* grant these roles permissions to cto schema in this database */
set @sql = 'IF ''?'' NOT IN (''master'', ''model'', ''msdb'', ''tempdb'') ' +
' begin ' +
' use ? ' +
' grant execute on schema :: ' + @dbschema + ' TO dbs_execute ' +
' end '

print (@sql)

exec master.dbo.sp_MSforeachdb @sql

set @sql = 'IF ''?'' NOT IN (''master'', ''model'', ''msdb'', ''tempdb'') ' +
' begin ' +
' use ? ' +
' print ''?'' '+
' grant select on schema :: ' + @dbschema + ' TO dbs_select ' +
' end '

print (@sql)

exec master.dbo.sp_MSforeachdb @sql

set @sql = 'IF ''?'' NOT IN (''master'', ''model'', ''msdb'', ''tempdb'') ' +
' begin ' +
' use ? ' +
' grant delete on schema :: ' + @dbschema + ' TO dbs_delete ' +
' end '

print (@sql)

exec master.dbo.sp_MSforeachdb @sql

set @sql = 'IF ''?'' NOT IN (''master'', ''model'', ''msdb'', ''tempdb'') ' +
' begin ' +
' use ? ' +
' grant insert on schema :: ' + @dbschema + ' TO dbs_insert ' +
' end '

print (@sql)

exec master.dbo.sp_MSforeachdb @sql

set @sql = 'IF ''?'' NOT IN (''master'', ''model'', ''msdb'', ''tempdb'') ' +
' begin ' +
' use ? ' +
' grant update on schema :: ' + @dbschema + ' TO dbs_update ' +
' end '

print (@sql)

exec master.dbo.sp_MSforeachdb @sql

set @sql =
'IF ''?'' NOT IN (''master'', ''model'', ''msdb'', ''tempdb'') ' +
' begin ' +
' use ? ' +
'if not exists( select u.name from master..syslogins l right join ' +
+ '?.dbo.sysusers u
on l.sid = u.sid
where u.name=''' + @newuser + ''') create user ' + @newuser
+ ' end '
print (@sql)
exec master.dbo.sp_MSforeachdb @sql



/* assign user to roles in each db */
set @sql = 'IF ''?'' NOT IN (''master'', ''model'', ''msdb'', ''tempdb'') BEGIN ' +
' use ? ' +
' exec sp_addrolemember ''dbs_execute'', ' + @newuser +
' end '

print @sql

exec master.dbo.sp_MSforeachdb @sql


set @sql = 'IF ''?'' NOT IN (''master'', ''model'', ''msdb'', ''tempdb'') BEGIN ' +
' use ? ' +
' exec sp_addrolemember ''dbs_select'', ' + @newuser +
' end '
print (@sql)
exec master.dbo.sp_MSforeachdb @sql

set @sql = 'IF ''?'' NOT IN (''master'', ''model'', ''msdb'', ''tempdb'') BEGIN ' +
' use ? ' +
' exec sp_addrolemember ''dbs_insert'', ' + @newuser +
' end '
print (@sql)
exec master.dbo.sp_MSforeachdb @sql


set @sql = 'IF ''?'' NOT IN (''master'', ''model'', ''msdb'', ''tempdb'') BEGIN ' +
' use ? ' +
' exec sp_addrolemember ''dbs_update'', ' + @newuser +
' end '
print (@sql)
exec master.dbo.sp_MSforeachdb @sql


set @sql = 'IF ''?'' NOT IN (''master'', ''model'', ''msdb'', ''tempdb'') BEGIN ' +
' use ? ' +
' exec sp_addrolemember ''dbs_delete'', ' + @newuser +
' end '
print (@sql)
exec master.dbo.sp_MSforeachdb @sql

set @newuser='test'
set @dbschema = 'dbs'

set @sql = 'IF ''?'' NOT IN (''master'', ''model'', ''msdb'', ''tempdb'') ' +
' begin ' +
' use ? ' +
' declare @s varchar(255) ' +
' set @s = ''alter user ' + @newuser + ' with default_schema = ' + @dbschema + ' '' ' +
' exec (@s) ' +
' end '


print (@sql)

exec master.dbo.sp_MSforeachdb @sql

-- Remove temporary tables
drop table ##orphans, ##objnames

The statement did not return a resultset - bug: MS JDBC driver + SQL Server 2000

There is a bug that has been driving me insane. I thought I was doing something wrong but apparently it's a bug in Microsoft SQL Server JDBC driver when using SQL Server 2000.

I had a bunch of sprocs that were failing for no apparent reason. I thought it was code change to incorporate a database connection pool. Coincidentally I updated the Microsoft JDBC driver at the same time and forgot about it.

Oddly enough the fix is to add "SET NOCOUNT ON" to stored procedures and then things run as expected.

The statement did not resturn a ResultSet

The Resultset has No Current Row

If you get this error using Microsoft JDBC driver to return records as a ResultSet:

The ResultSet has no current row

ResultSet rs = getResultSetFunction();
rs.next() <-----------

"ResultSet has not current row" does not mean that the resultset has not returned any records. It means that the recordset has not been positioned to the row you want to read from. When you first load the recordset, it's not positioned to any row.

So rs.next() will position the resultset to the first row (i.e. make the first row the current row), so you can read some data from that row.

The result set needs to have a current row to read some data from it.

Java Map Collection Classes

Choosing the right collection class has an impact on the performance of your application. In a previous post I gave a run down of Big O Notation. A hash map look up will have better performance generally than other types of collections when you can look up the data based on a key - O(1) since the underlying data structure is an array - and is useful when the order of the data is not important.

This article has a good run down of Java Maps including how they work and how to choose the right one for your application:

Java Map Collection Classes

In general the concept of optimizing for performance after the application has been built - by actually profiling the application and then making specific implementation decisions - is something I liked to hear. You need to understand the basics to implement a solid architecture but true optimization happens after you can measure the performance, taking all application variables into account from network to hardware to various software components and data distribution, and determining where your bottlenecks lie once you can actually measure the results of your particular application.

AJAX Attack Surface

I have read articles stating Ajax *supposedly* does not increase the attack surface of a web application.

Whether or not that statement is true, I've personally been cautious about integrating Ajax into web applications I've written. This isn't because I have anything against Ajax or think people shouldn't use it or that I think there's anything wrong with the technology - the concept is great. In fact I implemented an Ajax like client side very complex quote calculation JavaScript library of sorts for a web site prior to ever hearing of Ajax many years ago to reduce round trips to the server and improve performance for web visitors.

The reason for my caution is that I wanted to completely understand how Ajax functions -- and how to secure applications using it -- before integrating it. To prevent lost sales and problems for my customers, it is easier and just as effective to just leave it out for most web sites which don't have a good justification for using it. For web sites that perform very well, get great search engine rankings and drive lots of business to those customers (and isn't that the point?) without it, Ajax may only be a source of new problems and expense that is not worth the cool factor.

However I have had customers who insist on Ajax and there are times where it does make sense - especially for applications with complex user interfaces that might otherwise use frames, for example, to prevent going back to the server for the same data over and over again - so have been revisiting and working with Ajax lately and researching the security issues surrounding it.

I just read an article by a "white hat" security web site claiming Ajax does not increase the attack surface of a web application. It has very high search engine rankings presumably because all the people who don't understand or care about the security implications of all the new "cool" technology they throw on their web sites are linking to it to prove to the world that Ajax doesn't pose any additional security problems because that's what someone wrote - without really understanding or thinking about the underlying architectural changes that result from using Ajax.

Well, it's probably a semantic argument because the point is, your web site needs to be secure and you need to constantly monitor both your web site and the security newsgroups and publications for new threats. And yes, using Ajax is probably fine and any web technology you choose to use needs to be implemented securely.

However I disagree that Ajax does not increase the attack surface on a web application. Here's why:

#1. Definition of attack surface:

Attack Surface

The attack surface of a software environment is the code within a computer system that can be run by unauthenticated users.

In other words, more code, components and entry points = greater attack surface.

#2. How you reduce attack surface (same article):

The basic strategies of attack surface reduction are to reduce the amount of code running, reduce entry points available...

#3. New object created by third party - Ajax requires Ajax request object which you don't need if not using Ajax. That means you've introducing a new component and more code - which increases the attack surface. Every new thing you add to a software system is a potential new security problem no matter where it originates. That is the nature of software and systems - not because any particular thing you add has inherent security flaws (a separate issue and some things are riskier than others).

#4. More code (most likely) to support Ajax server side. Chances are pretty good that incorporating Ajax into your code base introduces server side code to support smaller requests server side that handle updating subsections of pages.

#5. More client side code. When someone clicks on something that initiates an Ajax request they are running client side code to make the request. That code wasn't needed before Ajax.

#6. Another entry point. Code is going through client side requests handled initially by JavaScript now instead of just being a link clicked on in the browser that is parsed into a request by the browser and sent over. In addition to the browser parsing you now have some code written by the web developer that has to decipher the request and do the right thing with it, potentially check additional user input, and handle errors appropriately. This is an additional step that wasn't there without Ajax. Probably you have new entry points server side as well for more, smaller requests to the web server.

Besides all that...

The thing that I am most concerned about is JavaScript code running client side that, if errors out before gets to the web server, is an error you'll never know about unless the programmer captures the error and sends it back to the server. This is something many sites don't do - including some large company web sites - because if they were doing it correctly I would never see JavaScript errors. And I do all the time.

Client side errors (in the most case without complex error logging or if your error logging fails) won't be in your web server logs and is a clear weak point that hackers will look at to attack your web application in ways that can be hidden from your view. They don't have to cover their tracks in logs. The person using the web site and the person running it may never know anything went wrong.

For example, if a hacker was able to exploit your JavaScript client side to redirect customers making purchases on your site to an alternate site instead so they get the sale, if it all happened on the client machine you'll think the customer just dropped out during the purchase. You'll have no tracks on your web server, no fingerprints, no clues. The customer may have gone onto make a purchase and received what they bought so they never knew there was a problem either. The other thing is code can be introduced to execute client side functions behind the scenes that don't go to the web server and don't show the user that anything is going on.

So, I would say Ajax increases the attack surface, but even if that technically is not true due to some semantically technical, philosophical justification - Ajax makes applications more complex with trickier security problems (maintenance issues, and bugs). In my opinion.

That doesn't mean I'm not using it. I'm just trying to figure out how to use it correctly and think too many people throw doodads and cool new technology on their web sites without really understanding the implications of using it and how to implement it in a secure manner.

Files Altered On Web Request

Just experienced strangeness with JavaScript in browser which has various maintenance and security implications.

While testing a web site using Jetty 6.1.12 and Eclipse with Jetty plugin, I noticed that somehow magically after requesting a web page, my files were ending up with extra characters at the end and I wasn't editing them. This happened both with HTML files and JavaScript files. It looked like the file would get the last 150 or so characters at the end of the document copied and pasted to the end of it. I would fix the file. Open it up to verify my fixes were saved. Then I would start and run the application and the problem would happen all over again.

This is odd - The source files shouldn't get edited and saved to disk when I make request to the web site. What would be changing them - the browser? Jetty? Jetty Eclipse Plugin? The OS? Browser plugin? Virus scanning software? Something in network? This is where it gets tricky.

The source files shouldn't get edited and saved to disk when I make request to the local web site. That should never alter the source code in the original web site. The original source files were altered, not just the code shown by view source in the browser or cached JavaScript source.

I figured out a few of things:

1. Could fix the files (delete the extra characters) in WordPad, but NotePad wouldn't make the changes stick for some reasons. I also realized later that NotePad wasn't alerting me when it didn't save the file if locked. WordPad would tell me. I had to shut down the web server to change the files even after setting ofline storage to 0 in FireFox.

2. After fixing the files they would continue to get corrupted. I finally made them read only to try to resolve the problem and changed my logging to indicate any time the files changed or there was an access error to try to pinpoint what was changing them. Even when the files were read only they still got changed and no logs indicating how they were changed or what changed them.

3. JavaScript functions in HTML file and did not have issues and the page loaded consistently on a few tests (did not test extensively).

4. FireFox does not log an error if your JavaScript src file path is incorrect. It just tells you a particular function is missing.

5. I realized even though I thought I turned off Firefox caching was still caching JavaScript files. Closing and opening browser would ensure I was getting the right file.

6. At one point I noticed that I had two opening head tags instead of correctly closing the head tag. That may have caused errors - but shouldn't have caused my source files to get altered.

7. I started working offline to remove any issues related to things getting cached somewhere related to being on the network even though the web page I am requesting is localhost.

8. When I initially had this problem I checked in both IE and FireFox and the same thing seemed to be happening, but perhaps one of them was altering the file so when I checked in the other I got the same result.

I don't know which of the above actually solved the problem but I'm not seeing it anymore and don't have time to explore this further. Just posting it online in case the makers of all this software want to figure out what in the world caused this because having software rewrite your source files when a web page is requested - is a bit troubling.

Generic Ajax Request

Here's a generic way to create an AJAX request (AJAX code below being JavaScript):

function createRequest(){
 
 request=null;
 
 try{
  request = new XMLHttpRequest();
 }catch(tryMS){
  try{
   request = new ActiveXObject("Mxsml2.XMLHTTP");
  }catch(otherMS){
   try{
    request = new ActiveXObject("Microsoft.XMLHTTP");
   }catch(failed){
    request=null;
   }
  }
 }
 
 return request;
}

And a sample function that uses it:

function getPage(){
 
 try{
  
  request = createRequest();
 
  var url ="/some/page.html";
  
  request.open("GET", url, true);
 
  request.onreadystatechange=displayPage;
 
  request.send(null);
 
 }catch(e){
  alert(e)
 }
 
}


function displayPage(){
 try{
  if (request.readyState== 4){
   obj = document.getElementById('someDiv');
   if (request.status== 200){
    obj.innerHTML = request.responseText;
   }
  }
 }catch(e){
  alert(e + request.responseText);
 }
} 

Next on the reading list:

Ajax Security

Remove file association with .lnk files in Windows 2008

Why Microsoft allows you to change all .lnk files to be associated with Notepad in Windows 2008 (or any other OS for that matter) I'll never know. Obviously that is going to royally mess things up. It's easy to do also because I just did it - I actually couldn't see I was opening a .lnk file because it had a really long path and in Windows 2008 the interface is funky (don't really like it). I right clicked and chose to open with Notepad. I guess the box was automatically checked to open all files of that sort with Notepad (I didn't check it).

Great. Now all my icons on the whole system show Notepad. When I clicked on any short cut it opened the short cut file in Notepad and displayed gibberish instead of opening the related program. So I thought ok, I'll just go into file associations and restore to default. Nope. Not so simple. There's no option to unassociate the file or restore to default (that I saw at first glance).

Sheesh.

So I just guessed that this might be in the registry and because I live my life on the edge, I searched for the .lnk file association and found it. Actually don't live that dangerously because I compared the settings to another machine that was working and determined that I needed to delete the "OpenWithList" node. Then log out and log back in.

Ouilla.

Jetty 503 - Service Unavailable

This is the type of programming issue that could drive one insane.

For some reason, suddenly getting 503 error in Jetty running Jetty Maven Plugin in Eclipse. That's it. No other explanation. Servlet could not be initialized. No additional clues to be had. Logging is printing to screen. Nada of use.

Finally just recomplied the whole multi-node project. Got errors that certain directories could not be deleted. Could not delete the directories manually until I closed Eclipse and everything related to this thing.

Once I deleted the directories and re-installed all the nodes, it worked again.

error reading /root/.m2/repository/javax/jms/jms/1.1/jms-1.1.jar; error in opening zip file

So I'm working away minding my own business and suddenly code that has been compiling stops compiling without any changes to the configuration. Apparently Maven has decided it needs to download something different - or re-download something which has moved and suddenly generating errors.

Anyway the error:

error reading /root/.m2/repository/javax/jms/jms/1.1/jms-1.1.jar; error in opening zip file

Maven attempting to download missing files

I agree.

But anyway I figured out that log4J was the problem and removed it from my project since working on different logging mechanism. Why in the world has this been compiling for weeks and suddenly stops? Hmmm?

Additionally after I pulled out log4J I got a bunch of errors about java mail components missing. Apparently those classes must have been part of the log4j jar but I didn't look.

The implication being that my app was using the log4j mailer classes rather than the java mail jar I downloaded from Sun java web site in my web app.

I would look into this further but since I removed log4j from my app my work here is done.

Asynchronous HTTP Requests

Informative article on Asynchronous HTTP requests.

Asynchronous HTTP Requests

Ah...but custom protocol handlers...ugh.

Checking out Jetty asynchronous HTTP client...
Jetty Http Client - Asynchronous web requests

Another implementation for review...but this an asycnhronous page fetcher... Hmm....the point of asynchronous is that you don't wait for a response right? At least that was my goal...

Asynchronous Web Page Fetcher

Cold Fusion Number Precision Problem

From some tax documentation I am reading for Sales Tax Online web tax calculation component:

"You may encounter a precision problem for very large amounts when using the ColdFusion number type. Because this type has a very large exponential range, it necessarily sacrifices precision in the number of significant digits it can carry. This will not generally be a problem with amounts expressed in currencies that have only two significant fractional digits (e.g., US Dollars), however, foreign currencies can have as many as four significant fractional digits. A value with a large non-fractional portion could suffer a rounding error in the fractional digits."

Chinese Jar Missing

Hmm. What is this all about.

Problem processing jar entry
com/ibm/icu/impl/data/LocaleElements_zh__PINYIN.class

Suddenly started appearing when running Jetty plugin without me adding anything to my project in terms of dependencies, included classes, etc.

Also coincidentally the whole structure of my Eclipse menus changed as well. Not sure if related. I didn't do anything to update Eclipse. I assume this is all automagic - not sure how I feel about that.

Anyway apparently this error can be ignored (don't usually like that answer) because some jar scanner couldn't find a jar, but didn't stop execution.

Read about it here:
http://www-01.ibm.com/support/docview.wss?uid=swg1PK81656

This is the stack trace:

2010-05-03 23:49:59.202:WARN::Problem processing jar entry com/ibm/icu/impl/data/LocaleElements_zh__PINYIN.class
java.lang.ArrayIndexOutOfBoundsException: 48188
at org.objectweb.asm.ClassReader.readClass(Unknown Source)
at org.objectweb.asm.ClassReader.accept(Unknown Source)
at org.objectweb.asm.ClassReader.accept(Unknown Source)
at org.eclipse.jetty.annotations.AnnotationParser.scanClass(AnnotationParser.java:596)
at org.eclipse.jetty.annotations.AnnotationParser.access$000(AnnotationParser.java:43)
at org.eclipse.jetty.annotations.AnnotationParser$2.processEntry(AnnotationParser.java:576)
at org.eclipse.jetty.webapp.JarScanner.matched(JarScanner.java:152)
at org.eclipse.jetty.util.PatternMatcher.matchPatterns(PatternMatcher.java:82)
at org.eclipse.jetty.util.PatternMatcher.match(PatternMatcher.java:64)
at org.eclipse.jetty.webapp.JarScanner.scan(JarScanner.java:75)
at org.eclipse.jetty.annotations.AnnotationParser.parse(AnnotationParser.java:588)
at org.eclipse.jetty.annotations.AbstractConfiguration.parseWebInfLib(AbstractConfiguration.java:108)
at org.eclipse.jetty.annotations.AnnotationConfiguration.configure(AnnotationConfiguration.java:77)
at org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:975)
at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:608)
at org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:155)
at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:349)
at org.mortbay.jetty.plugin.JettyWebAppContext.doStart(JettyWebAppContext.java:102)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:55)
at org.eclipse.jetty.server.handler.HandlerCollection.doStart(HandlerCollection.java:165)
at org.eclipse.jetty.server.handler.ContextHandlerCollection.doStart(ContextHandlerCollection.java:162)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:55)
at org.eclipse.jetty.server.handler.HandlerCollection.doStart(HandlerCollection.java:165)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:55)
at org.eclipse.jetty.server.handler.HandlerWrapper.doStart(HandlerWrapper.java:92)
at org.eclipse.jetty.server.Server.doStart(Server.java:231)
at org.mortbay.jetty.plugin.JettyServer.doStart(JettyServer.java:69)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:55)
at org.mortbay.jetty.plugin.AbstractJettyMojo.startJetty(AbstractJettyMojo.java:433)
at org.mortbay.jetty.plugin.AbstractJettyMojo.execute(AbstractJettyMojo.java:377)
at org.mortbay.jetty.plugin.JettyRunMojo.execute(JettyRunMojo.java:546)
at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:105)
at org.apache.maven.lifecycle.DefaultLifecycleExecutor.execute(DefaultLifecycleExecutor.java:577)
at org.apache.maven.lifecycle.DefaultLifecycleExecutor.execute(DefaultLifecycleExecutor.java:324)
at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:247)
at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:104)
at org.apache.maven.cli.MavenCli.execute(MavenCli.java:427)
at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:157)
at org.apache.maven.cli.MavenCli.main(MavenCli.java:121)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:290)
at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:230)
at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:409)
at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:352)

Referencing Jar from Maven not in local Repository

If you want to use a jar but not install it in your local Maven repository this seems to work:

Set scope to system. Then point to the jar in systemPath element and pretty much make up the rest.

<dependency> 
    <groupId>com.bmetrix</groupId> 
    <artifactId>yeehaw</artifactId> 
    <version>0.1.0</version> 
    <scope>system</scope> 
    <systemPath>C:\some\crazy.jar</systemPath> 
</dependency> 

JDBC Connection Pool - Minimalist Options

Revisiting database connection management. I took a look at C3P0 which is an open source JDBC driver and looks pretty simple to use.

The only thing I don't like about downloading some of these libraries is the amount of code contained in them I don't really need. I looked at the source and don't need anything for JBoss, for instance. I prefer simplicity and minimalist code, personally. Just to get some functionality in Eclipse I have to download fifty bazillion classes. Yes, bazillion.

And have some other personal reasons for not wanting to use all the J2EE stuff for managing resources. It's my code, so I don't have to conform to the masses in this case. When developing for another company I would probably have to follow Java law.

So anyway I found this interesting example as a starting point for a JDBC connection pool without all the extraneous stuff I don't need:

JDBC Connection Pool - Example Code

I'm sure this needs revisions and not completely what I need (already found it did not override a bunch of things now in the connection interface and is using out of data classes per some documentation I ran across) but maybe something between the two would be nice.

Maybe if components you download had a way to specify and get only the functionality you need when you get them instead of downloading every option under the Sun...no pun intended ;-)

Use Maven Jetty Plugin in Eclipse

I wanted to run Jetty in Eclipse so I can write code, debug, test, etc. Ran into some issues. Solution here.


Add Jetty plugin to your pom file:

<build>
  <plugins>
    <plugin>
     <groupId>org.mortbay.jetty</groupId>
     <artifactId>jetty-maven-plugin</artifactId>
    </plugin>
  </plugins>
<builds>

I then tried to run the jetty goal from the command line to keep it simple:

mvn jetty:run

No user specific settings.xml file

Then, something was griping because I never added a settings.xml file to my local repository. Copy the settings.xml file from the directory where you installed maven and drop it into your local repository (see my post on Java development environment if you need more info).

Missing Dependencies

Next...complaints about missing jars not in Maven central repository.

Download the jars I found searching in Google (what would we do without Google as nothing Java related seems to be well documented completely from end to end):

el-impl-2.2.jar

http://download.java.net/maven/2/org/glassfish/web/el-impl/2.2/

el-api-2.2.jar

http://download.java.net/maven/2/javax/el/el-api/2.2/

jsp-impl-2.2.jar

http://download.java.net/maven/2/org/glassfish/web/jsp-impl/2.2/

jsp-api-2.2.jar

http://download.java.net/maven/2/javax/servlet/jsp/jsp-api/2.2/

Install into maven repository with the following commands (make sure you're in the directory where you downloaded the jars or provide full path to jars):

mvn install:install-file -DgroupId=org.glassfish.web -DartifactId=el-impl -Dversion=2.2 -Dpackaging=jar -Dfile=el-impl-2.2.jar

mvn install:install-file -DgroupId=javax.el -DartifactId=el-api -Dversion=2.2 -Dpackaging=jar -Dfile=el-api-2.2.jar

mvn install:install-file -DgroupId=org.glassfish.web -DartifactId=jsp-impl -Dversion=2.2 -Dpackaging=jar -Dfile=jsp-impl-2.2.jar

mvn install:install-file -DgroupId=javax.servlet.jsp -DartifactId=jsp-api -Dversion=2.2 -Dpackaging=jar -Dfile=jsp-api-2.2.jar

PluginsGroup

Next I had to add this to my settings.xml file in my local respository settings.xml file:

<pluginGroups>
<pluginGroup>org.mortbay.jetty</pluginGroup>
</pluginGroups>

No webapp directory or start up file

I noticed that Jetty was referencing a non-existant path for the web app. Default is under your project /src folder in /webapp directory. I also added the default index page under my project src directory:

/src/webapp/index.html

Bug: Jetty context fails with spaces in path

Ok the next issue was that apparently if you have spaces in the path to your local repository Jetty will fail for that reason also.

Details are here: http://jira.codehaus.org/browse/JETTY-1166

So...I had to move my whole local repository to an alternate directory with no spaces. I uncommented the local repository setting in the settings.xml file and changed the path to the new location.

<!-- localRepository
| The path to the local repository maven will use to store artifacts.
|
| Default: ~/.m2/repository-->
<localRepository>/path/to/local/repo</localRepository>

Finally...Jetty runs. If you got this far test it out by going to this url on your local machine:

http://localhost:8080

Running from within Eclipse

Okay that's nice but I wanted to run Jetty from within Eclipse. I right clicked on project pom.xml file, chose "Run As" then "Run Configurations". Double clicked on "Maven Build" node to add a new Maven run configuration. In the dialog box I typed a name - called it "run jetty". I entered the path to the directory containing my pom.xml file as the base directory. For goals I entered "jetty:run". I checked Debug Output, update snapshots and resolve workspace dependencies - clicked Apply, then run.

Yippee. Jetty is running in Eclipse and you can see the output in the Eclipse console.

SQL Server Permissions

Had an interesting issue after restoring a database to SQL Server 2008 database server. None of the users could select from any tables. When restoring a SQL Server database, you pretty much always have to go in and recreate the accounts and add them to the databases again with the appropriate permissions. This was a bit different and odd.

Here's how I solved it:

1. Tried adding a select role which had select permissions on the database (did not solve it but here's now to do it):


Let's say your user name is "dbuser" and you want to create some roles to do different things like execute, insert, select, etc. You could do something like this - which is probably overkill and you might want to set up a different schema also - but I was just testing anyway:

/* CREATE ROLES */
CREATE ROLE db_execute
CREATE ROLE db_select
CREATE ROLE db_delete
CREATE ROLE db_insert
CREATE ROLE db_update

/* GRANT PERMISSIONS TO ROLES */
GRANT EXECUTE ON SCHEMA :: dbo TO db_execute
GRANT SELECT ON SCHEMA :: dbo TO db_select
GRANT DELETE ON SCHEMA :: dbo TO db_delete
GRANT INSERT ON SCHEMA :: dbo TO db_insert
GRANT UPDATE ON SCHEMA :: dbo TO db_update

/* ADD USER TO ROLES */
EXEC sp_addrolemember 'db_execute', 'dbuser'
EXEC sp_addrolemember 'db_update', 'dbuser'
EXEC sp_addrolemember 'db_insert', 'dbuser'
EXEC sp_addrolemember 'db_delete', 'dbuser'
EXEC sp_addrolemember 'db_select', 'dbuser'

2. You could grant permission to all tables and views in your database like this handy dandy script I found here

use MYDBNAME

DECLARE @login varchar(50)
SET @login = 'dbuser'

DECLARE @tables TABLE(ROWID int IDENTITY(1,1), SQLSTR varchar(500))
INSERT INTO @tables
SELECT 'GRANT SELECT ON ' + NAME + ' TO '+@login
FROM sysobjects
WHERE TYPE = 'U'
AND NAME NOT LIKE 'SYNC%'

DECLARE @rowid int, @sqlstr varchar(500)
SET @rowid = 0
SET @sqlstr = ''
DECLARE grant_tbl_cursor CURSOR FOR
SELECT ROWID, SQLSTR
FROM @tables
ORDER BY ROWID
OPEN grant_tbl_cursor
FETCH NEXT FROM grant_tbl_cursor
INTO @rowid,@sqlstr
WHILE @@FETCH_STATUS = 0
BEGIN
EXECUTE (@sqlstr)
FETCH NEXT FROM grant_tbl_cursor
INTO @rowid,@sqlstr
END
CLOSE grant_tbl_cursor
DEALLOCATE grant_tbl_cursor

DECLARE @views TABLE(ROWID int IDENTITY(1,1), SQLSTR varchar(500))
INSERT INTO @views
SELECT 'GRANT SELECT ON ' + NAME + ' TO '+@login
FROM sysobjects
WHERE TYPE = 'V'
SET @rowid = 0
SET @sqlstr = ''
DECLARE grant_vw_cursor CURSOR FOR
SELECT ROWID, SQLSTR
FROM @views
ORDER BY ROWID
OPEN grant_vw_cursor
FETCH NEXT FROM grant_vw_cursor
INTO @rowid,@sqlstr
WHILE @@FETCH_STATUS = 0
BEGIN
EXECUTE (@sqlstr)
FETCH NEXT FROM grant_vw_cursor
INTO @rowid,@sqlstr
END

3. But what really solved the problem for me was looking at the effective permissions on the table that I initially tested. Right click, choose properties and check out the effective permissions tab. I noticed that the public role was denied pretty much any permission on any database object. Given that all users must be in the public role and cannot be removed, had to fix that.

First I verified that these permissions were also returning from a query on the permissions and users for that object:

select OBJECT_NAME(major_id) as object,
USER_NAME(grantee_principal_id) as grantee,
USER_NAME(grantor_principal_id) as grantor,
Permission_name,
state_desc, class
from sys.database_permissions
where major_id=OBJECT_ID('Table_In_Question')

Sure enough, public had deny set on just about everything.

So...I undenied it but removing all permissions from the public role and resetting permissions as desired.

REVOKE SELECT FROM PUBLIC

It's a little odd that we are revoking the denying of something, but that's how it works. Running my permissions query after this showed that the public user had no object level select priveledges on that table - neither grant nor deny. I tested and was able to select from the table with the other user that was part of the public role.

I also tested after my changes and that same user was not able to change permissions or table strucures (which is what I was after in the first place via more granular database permissions.) Further review of system tables and probably schema changes are in order, but this particular problem is solved.

Run Batch Files with Cold Fusion

For some processes that use batch files to execute scripts related to processing builds, database imports, data syncs, etc. We also want to allow users who don't have access to the server remotely via remote desktop to be able to execute these commands but also log who executed them when and send out notifications. By surrounding these processes with a secure UI we can allow users to execute these functions without giving remote desktop access to the server, require authentication and track who executed what, when. ColdFusion happens to be the language available in this environment, and this is the ColdFusion tag that runs a batch file.

<cfexecute
name="ApplicationName"
arguments="CommandLine Arguments"
OUTPUTfile="Output file name"
timeout="Timeout interval in seconds">

With code such as this will be important to ensure that only the specified scripts can be executed by limiting user accounts that can run this and against what files and directories.

Backup and Restore SVN Repository

Basic command for backing up SVN repository:

svnadmin dump repositorypath > backupname.svn

More on backups at:

Back up SVN Repository

To restore the backup on another machine:

svnadmin load respositorypath < backupname.svn

Truncate SQL Server Transaction Log

USE DatabaseName
GO
DBCC SHRINKFILE(<TransactionLogName>, 1)
BACKUP LOG <DatabaseName> WITH TRUNCATE_ONLY
DBCC SHRINKFILE(<TransactionLogName>, 1)
GO

Transaction logs are truncated on backup or checkpoints depending on the model you're using:

http://technet.microsoft.com/en-us/library/ms189085.aspx

However the backup will include the larger transaction log so you'll want to back up after truncation to get a smaller backup file and not have to restore the entire transaction log.

Role Management Tool in Windows 2008

If you search for "Role Management Tool" for Windows 2008 on Microsoft.com, chances are you won't find what you seek.

If you go to Google and type in that phrase you'll probably find the first site that comes up to be exactly what you are looking for.

I do not know why Microsoft makes things so different between each version of their software, but they do. The Role Management Tool allows you to add features to Windows such as .NET 3.5 which ships with Windows 2008 but apparently is not installed by default.

I like the idea about not having everything you don't neccessarily need and want installed and running, but I found the error message when I tried to download and install .Net 3.5 service pack 3 a bit cryptic. Why not just pop up a link to instructions that explain how to install it?

And why doesn't Microsoft search return a relevant answer that explains what the Role Management Tool is in Windows 2008?

Mysteries of life to ponder....

Java Development Environment for Windows

Since I'm setting up Java software development Environment, thought I would document parts of the process for any potential new Java programmers out there trying to get set up on Windows. Not going to go into every detail...but this would get someone started with commonly used software tools for Java software development and some tips for Windows users.

A few things before we get started:

Some of the instructions below indicate that you should open a command prompt window or type a command. You can get to a command prompt window by going to the start menu, choose run, type cmd and hit enter.

Additionally there are instructions for setting environment variables. To get to those go to Control Panel, click on System, Advanced and Choose Environment Variables. These instructions differ slightly on different versions of Windows but should be pretty close.

When you run Java code it can run using a JRE (Java Runtime Environment) or JDK (Java Development Kit). The JRE is what runs in your browser. It is for people who need to run but not write Java code. The JDK has additional functions for writing and compiling Java code. When configuring development tools, make sure you reference the folder where you download and install the JDK, not folders containing a JRE.

Java

Download java - on the Sun web site choose downloads, SE and the JDK (not
E)

java.sun.com

Run the executable (exe file) you just downloaded to install Java.

Set the JAVA_HOME environment variable:

Go to Control Panel, click on System then Advanced, then Environment Variables. Click new. Type JAVA_HOME for the name of the variable and the path to the directory you stalled java into for the value. For example:

C:\Program_Files\java\jdk1.x.x_xx

Note: the directory you enter sould be the one that contains the \bin folder. In other words for the example above if I look in the directory above I will see a subfolder named "bin".

Eclipse

Eclipse is an "IDE" or program used to write and compile Java code (among other things). Eclipse integrates with a lot of tools and applications. I have found Eclipse used at almost every company I've worked at recently for Java software projects (and some other types of software as well):

Eclipse

Click on downloads and choose one of the options for Java programmers. Either one will work fine to get started but if you know you're using J2EE then you can choose the larger download. Eclipse is Java software so it runs on any platform so you don't need to worry about getting the right version for Windows, Linux, etc. Just download the file, right click to unzip and choose the folder where you want the program files to end up. After unzipped, there will be an "eclipse.exe" file in the folder where you unzipped the files. Double click on it to start the program. You can also right click on it and choose to create a shortcut which you can drop on your desktop, quick launch toolbar or drag to your start menu.

Subversion

Subversion is a "version control" system. It tracks versions of your files so you can go back to previous versions in case of error. It also helps companies track which files were released to production servers so they know which versions of files are in production at any given time, and roll back to a previous successful release in case a new release causes a problem - regardless of what changes happened in between releases.

Subversion is an open source product and the source code can be downloaded from the Apache web site at:

http://subversion.apache.org

If you're on Windows you may not want to go through the build process to compile the software in order to run it - it's a little complicated. Binaries (or the files used to install and run the program on Windows) are offered on the Apache site. Collabnet's version claims to be the "certified" binaries. Always take a look at who writes and supports the software you download and where they are located when considering your options. I took a look around the CollabNet site and saw that Tim O'Reilly of O'Reilly & Associates is on the Board of Directors...figured that was a good sign.

The CollabNet version of subversion can be found here:

Subversion Binaries

Choose to run subversion as a service and it will run every time you start your computer. You can open up Windows services and set it to run manually as well. You'll choose a port on which to run the software - I chose not to run on port 80 (standard web port) so it wouldn't interfere with other things I have running on that port. You can choose a different port, say 81 or 8888 or whatever suits your fancy but doesn't conflict with other software on your machine. Typically you'll run subversion on a server but you can also run it on the same machine on which you develop if you don't have any other option.

Cannot load C:/Program Files/CollabNet/Subversion Server/httpd/modules/mod_dav_svn.so into server

Unfortunately when I opened up Windows Services (Control Panel / Administrative Tools / Services) and tried to start the two CollabNet services, the Apache service would not start. The error in the Windows Application Event Log (Control Panel / Administrative Tools / Event Logs) was:

The Apache service named reported the following error:

>>> httpd.exe: Syntax error on line 116 of C:/Program Files/CollabNet/Subversion Server/httpd/conf/httpd.conf: Cannot load C:/Program Files/CollabNet/Subversion Server/httpd/modules/mod_dav_svn.so into server: The specified module could not be found.

The service basically starts the Apache web server so I manually started the Apache web server by going to the folder in which I installed CollabNet \Subversion Server\httpd\bin\httpd.exe and running that file. That seemed to work fine. However I wanted Subversion to run as a service...

I read in some blog posts that restarting Windows might help...that did the trick.

Test SVN is running

To see if SVN is up and running I went to a command prompt (start / run / type "cmd" and hit enter) and type

svnadmin help

There I get the list of commands available from SVN admin. Seems all good.

Create SVN Repository

To create a new repository (the place where subversion saves your file revisions but in a special format that should only be accessed through subversion commands and tools), I navigate to the place where I want the repository to be. Then type:

svnadmin create myrepo [or whatever you want your repository to be named]

Ouila - repository created.

Running SVN under separate Windows account

You'll want to run this server under it's own user name for a secure environment. Set up a new user account and give it appropriate permissions. I had to use administrator over user but would be preferable to not run as a full-fledged admin or limit the permissions on files for this account to only what is required for proper functioning. Then right click on the service and change the login as to login with the user/name and password you created for this service. That way whatever actions this software takes when running will show up in the logs as "svn" rather than some generic account. This helps track down problems and setting up restrictive permissions for this user will limit potential security problems.

Error 1053 with SVN Serve Service

Another note - started getting error 1053 when I deleted the initial repository directory I configured for SVNServe. If that happens, you can [very carefully] edit Windows registry and change the path used by the service from the directory you initially set up to whatever the valid directory is that this user account has permissions to access. You can see what directory the service is using for the repository by right clicking on the service and looking at the start up information, then search for that in the registry.

SVN Authentication

You probably want to set up user names and passwords so not just anyone can access your SVN Server. There are multiple ways to do this but here's a simple way to start:

Edit the httpd.conf file (search for it or find under apache files where you installed Subversion)

There's a few lines at the end starting with:

<Location /svn>
....

Edit that section as follows:

SVNParentPath should be the folder that contains all your repositories. You can also set up separate locations for each repository using SVNPath instead of SVNParentPath if you need to configure your repositories separately.

You'll add the path to the file that contains users (AuthUserFile) and set a few other options. Read up on this further if you want to know more about all the options.

<Location /svn>
   DAV svn
   SVNParentPath C:\path\to\folder\containing\repositories
   AuthType basic
   AuthName "welcome to my repository"
   AuthUserFile C:\path\to\users.txt
   <LimitExcept GET PROPFIND OPTIONS REPORT>
      Require valid-user
   </LimitExcept>
</Location>

The above options create read only access to the repository.

You can remove LimitExcept line to give a user full access or change the LimitException options as appropriate for your needs.

Restart Apache service.

That works but the CollabNet documentation says that to ensure you're not passing SVN passwords plain text over the network you'll want to use digest mode. That has some slight variations and more information can be found in Apache documentation - but here's their example which worked for me:


<Location /svn>
DAV svn
SVNParentPath /var/svn
AuthType Digest
AuthName "Subversion repository"
AuthDigestDomain /svn/
AuthUserFile /etc/svn-auth-file
Require valid-user
</Location>
Now we'll need to add a user.

First of all, I had a problem running this command below because the bin folder for Apache was not on my Path. Add that by going to your Control Panel, click System, then the Advanced tab. Very carefully edit the one that is named "PATH" in the list of variables. Add the path to the folder located in the folder in which installed subversion end with /httpd/bin.

For instance if you installed subversion in C:/SVN add C:/SVN/httpd/bin to the end of your PATH environment variable.

Hint: I always copy and paste the path so I don't make a typo...

If there are other things already listed for your PATH variable, which I'm sure there are, precede the following line with a semi-colon (;) if there isn't one there already. Each item on the path is separated with a semi-colon. hit the Apply button and close the window.

Also if you have a command prompt window open close it and open a new one to get the updated path. You can check what the PATH is in your command window and make sure your changes took effect by typing:

set PATH

Navigate to the folder where you specified your users file would be stored in your SVN configuration above.

Type:

htpasswd -c users.txt [username_here_no_brackets]

You'll be prompted to enter a password.

Woop-ti-doo. User created.

Note that if you have a large number of users you will want to consider other options such as storing user names and passwords in a MySQL database. Searching text files can lead to performance issues.

Tortoise SVN

Once you have a version control system set up you open up a command prompt window and type commands to save versions of your files, however it's a lot easier to use Tortoise SVN on Windows. Tortoise SVN provides a nice user interface that allows you to just go to any directory on your machine, right click, enter the URL to your repository and choose "checkout" to get the files in the repository for the first time. You can also easily add new files to your repository by right clicking and choosing "add".

Tortoise SVN can be downloaded here:

Tortoise SVN

Note: Make sure you bypass the ads and scroll to the bottom of the screen to get the correct links.

Prevent Password Caching

One other thing I like to do is turn off password caching so they are not saved on the server or my client machine. To do that the book I have says to change store_passwords=no in the subversion config file -- found with your Windows user account files which on XP would most likely be c:/Documets and Settings/[Your User Name]/Application Data/Subversion but might be different in another scenario).

However I found the following documentation within that file related to this parameter - looks like it gets set in the servers file in the same directory now:

### The rest of this section in this file has been deprecated.
### Both 'store-passwords' and 'store-auth-creds' can now be
### specified in the 'servers' file in your config directory.
### Anything specified in this section is overridden by settings
### specified in the 'servers' file.

Add files to the repository

OK so now it's time to add files to the repository. The typical structure for a project in svn is:

/project
    /branches (branches off the main code folder ...read up on this one...)
    /tags (snapshots of specific revisions of the code in time - for builds, etc)
    /trunk (the main folder where the current code goes)

For more on structuring repositories...read up.

I found this book to be helpful for both Subversion and Tortoise SVN:
Practical Subversion

In order to create a new project in the repository I first created the folder where I wanted my project to live. I set up the three standard folders (tags, branches and trunk). Then right click on the folder ABOVE the top folder I want to go in subversion. Everything below will get added. Choose "subversion" then "import". Enter the URL to the repository. Hit OK.

Note - if you're trying to figure out what the url to your repository should be on your local machine, right click on the folder where your repository is stored and choose "Tortoise SVN / Repo browser". This will pop up the repository browser with the URL in the dialog box. You can just copy it and use it when you right click on your project folder to checkout, add, import and/or commit files.

And from this point on....I leave the rest to your further reading and exploration.....

Maven

Maven is a tool that is used to help "build" or compile Java programs, tracking all the dependencies in a project. It can also help with automated testing, deployment and other aspects of managing software projects.

Maven can be downloaded and installed from:

Maven

For Windows you'll want to download the pre-compiled (binary) version unless you want to compile the software youself and the zip format of the file which will look something like this:

apache-maven-x.x.x-bin.zip

Unzip the files to a location of your choice such as:

C:/Progam Files/Maven

Once unzipped you should see within the above directory a subdirectory called /bin. In other words:

C:/Program Files/Maven/bin

You'll want to add that bin directory to your PATH environment variable as described above for SVN. Find system information, advanced, environment variables. Edit the PATH environment variable and add ;C:/Program Files/Maven/bin (or whatever the path is to the /bin directory where you unzipped the files).

You'll also need ot add a new environment variable. Click the add button. Enter M2_HOME as the variable name and enter the directory that contains the bin folder as the value for the variable (i.e. for example above C:/Program Files/Maven).

Test that Maven is running by going to a command prompt (start/run type cmd and hit enter) and type:

mvn -v

This should print out the Maven version on the screen - ok Maven's running.

Alrighty then, let's create a maven project. Then we can add code to this project and use Maven to build it, test it, generate documentation, etc.

Navigate via command prompt to the folder where you want to create the project. Type something like the following, replacing the Group, Artifact and Package with whatever you want those to be. Group and Artifact identify the project so you can refer to it later. You can read more about those. Package is the Java package that the code will be in - you can read up on Java packages if you're not familiar with those. And by the way "archetype" is a fancy word for a template. So when we generate this project it's based on the standard Maven template for a project - the files and directories that are created when you run this command are based on the default Maven project template or archetype.

mvn archetype:create -DgroupId=com.somecompany.somegroup -DartifactId=component -DpackageName=com.somecompany.xyz

NOTE: Case sensitive. No extra spaces between names, equal signs, and values. No line breaks (which may throw an error like 'xyz is not a recognized internal or external command)

The first time you run this you'll notice Maven downloading a bunch of dependencies it needs to run. This is what Maven does - tracks the dependencies for software projects and grabs them from specified repositories as needed. The files will be downloaded to your local Maven repository the first time and accessed from there after that point. Your local repository on Windows is by default in the typical place Windows dumps your files in an .m2 folder (C:/Documents and Settings/your_name/.m2/repository on XP). This is configurable if you want to change it.

There are advanced ways to configure Maven to only work with certain repositories such as companies that only want employees downloading components from internal locations that have verified, secure versions of code and also minimize Internet network traffic...you can read up on that.

After running the command to create a new project, you can list the files that were created and you'll see the default project structure.

The pom.xml file in the root directory for the project is the file that you will configure to tell Maven what to do with your project. Open this file to see the default configuration that was set up. Hint: Opening with Notepad might not give you the correct line breaks. Use WordPad, IE, etc.

If you navigate within the project to the /src/main/..../App.java file this is the default main class that Maven created for your project. Obviously you can change this as needed.

OK now you have a Maven project. You can use this as a starting point to add code, compile it, run tests, configure plugins, generate documentation, etc.

To compile and put the resulting output into your local repository, you simply type:

mvn install

Maven Dependencies

Look at the pom.xml file that was generated and you'll see a dependency listed - JUnit. More about JUnit later. When Maven builds your project, it will go out and get this dependency and bring it down to your local repository. If your code requires other .jar files to run, you can add them as dependencies to your project following the example for JUnit. When you run "mvn install" Maven will automatically download the dependencies you have specified from the Maven centra repository if they do not exist in your local repository.

In order to find a dependency Maven uses the artifact id and other information in the configuration for that dependency such as version number. So you'll need to get the right information for the particular dependency you're adding to your project.

Dependencies not in Maven Repository

Sometimes you'll find a jar file you want to use is not in the central repository for Maven. In other words whomever created hasn't added it to or doesn't allow it to be added to the main repository managed by Maven so you can download it automatically by just adding the configuration to your pom file. In this case you'll have to install it into your own repository manually. Once it's in your repository you can add it to pom files to include it in projects.

To do this use mvn install:install-file command.

For example if you want to include the Microsoft SQL Server JDBC driver in your project you'll need to download the latest jar file from Microsoft. Open a command prompt and navigate to the directory where you downloaded this jar. If you've downloaded version 3.0 which is sqljdbc.jar type the following command.

mvn install:install-file -Dfile=sqljdbc.jar -DgroupId=com.microsoft.jdbcdriver -DartifactId=sqljdbc -Dversion=3.0 -Dpackaging=jar

Once you've done that you can include this jar in the pom file for a project like this:

<dependency>
<groupId>com.microsoft.jdbcdriver</groupId>
<artifactId>sqljdbc</artifactId>
<version>3.0</version>
</dependency>


Maven Plugins

Maven has a bunch of plugins that can perform various build tasks for you. In order to use a plugin you add it to your pom file. For example if I want to use the SCM plugin which integrates with our source control software (Subversion) I would add it to the default pom.xml that was created above, putting the following at the end before the closing <project> tag:

<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins
<artifactId>maven-scm-plugin

<configuration>
<!-- your example configuration here -->
</configuration>
</plugin>
</plugins>
</build>

When you type mvn intstall, that plugin will be downloaded if not already in your repository.

NOTE: If you need a specific version you can add that to the configuration. If you get an error that a plugin or dependency could not be downloaded, you can remove the version number to get the version that is available.

You can find a list of Maven Plugins here:

http://mvnrepository.com

Multi-Node Projects

If you want to include childe poms within a parent pom add a modules section to your parent pom file that lists the other pom artifact ids as modules. This way you can have maven run against multiple projects/poms in one step. For instance you might have a library (jar) that contains core classes used in multiple projects with artifactId as "project1" in its pom.xml file. Then you have a library containing the classes specific to the project you are working on with artifactId "project2" its pom file. You want to build both in one step so you create a parent pom and include these additional modues as follows:

<modules>
<module>project1</module>
<module>project2</module>
<modules>

Maven Plugin for Eclipse

Ok you're a Windows user so probably rather not deal with this command line stuff right? Well, then download the Maven Plugin for Eclipse which provides a UI to manage Maven through Eclipse. I found the instructions at the link below to be out of data for the latest version of Eclipse but will tell you more about M2Eclipse:

Maven Plugin for Eclipse

In the version of Eclipse I am using, I had to go to the help menu and choose "Software Updates", click on "Available Software" tab, then "Add Site" button. Then enter "http://m2eclipse.sonatype.org/sites/m2e" in the dialog box that pops up (no quotes) and click "OK". You should see the Sonatype site added to the list of software repositories in the install software dialog box. Check it and click the "Install" button. Follow the steps to install the plugin.

Ok now you can create a new Maven project in Eclise. Choose File | New | Other | Maven Project. This will allow you to generate a new project based on a Maven archetype.

You can also choose to open one the existing project you generated above using the create:archetype command by choosing File | Import | Maven | Existing Maven Projects and choose the directory that contains the pom.xml file for your existing project.

The first time I ran the Eclipse plugin I got an error:

"Maven plugin requires Eclipse to run in JDK...etc."

To fix I did the following:

1. Add the jvm to your eclipse.ini file found in the directory in which you installed Eclipse. Add this line with the correct path to the Java version and install directory you specfied above when you installed Java:

-vm
C:\Java\JDK\1.5\bin\javaw.exe

I noticed that you must put this on two different lines - if you put those two things on the same line it fails. Additionally it worked for me when I inserted it at the beginning of the eclipse.ini file.

I've seem comments in blogs that you'll need quotes around a path with spaces in it, like this:

-vm
"C:\Program Files\Java\JDK\1.5\bin\javaw.exe"

However I found this not to be true and caused an error on one machine I tried this on.

More on eclipse.ini: Eclipse ini

2. In Eclipse choose Window | Preferences. Click (+) next to Java. Click installed JREs. Click Add button. Choose standard vn and Next button. Choose the same directory you added as JAVA_HOME when you installed Java above. Click Finish. I also removed the older version of the JRE since I don't want to end up using it by mistake. For my purposes I just use the latest version of the JDK.

3. I noticed at this point still had a yellow warning about the JDK version in Eclipse for my project referencing 1.4 when 1.4 was not installed. To fix I right clicked on the project node, selected "Build Path", "Configure Build Path", then clicked on the "Libraries" tab. I removed the reference to JDK 1.4. I clicked the "Add Library" button. I chose the option "JRE System Library" which selected the JRE I configured above in Maven (which is actually JDK, not JRE, but whatever).

Next error...apparently Maven uses JDK 1.3 by default. You'll need to tell Maven to use a higher version if you want to use generics or other newer features in higher versions of the JDK. Add the following to your pom.xml file for your poject. I added this after my dependencies node:

<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<configuration>
<source>1.6</source>
<target>1.6</target>
</configuration>
</plugin>
</plugins>
</build>


Maven Book

I found this book helpful for using Maven, however I also found new useful tools that replace plugins in Maven 2.2 that are not documented in this book...so read up on new features in Maven and plug ins online.

Maven: The Definitive Guide

JUnit

JUnit is used to write unit tests. These unit tests basically call various methods in your code and verify tha the code behaves as expected - throwing and catching errors where it should and returning correct values.

An example unit test class was created in the project above we created with the Maven archetype. You'll find this code in the /src/test/java directory. The calss is called AppTest.java. Test classes typically are named the name of the class they are testing plus "Test". you can see that within the class the JUnit classes that run tests are imported at the top. The class extends (read about abstract classes if not familiar) the class TestCase so it can run tests. There's a function that returns the whole suite of tests contained in the class (which happens to be one in this case) and a function that runs a test. You can add more tests to the class following the example. I'll leave it to you to read up on how to write tests.

To run the tests via maven you can run: mvn test

Test are also run as part of: mvn install

To run the tests from within Eclipse you can right click on the class or the particular test in the project explorer tree and choose "Run As" then "Junit Test".

You'll notice that in your pom.xml file the JUnit dependency has in the configuration:

<scope>test</scope>

That means this dependency is needed for test scope only and will not be included for deployment. In other words your final application that you deploy on the web or run as an application doesn't include the tests and the JUnit dependency.

Jetty

Jetty is a web server which is efficient, fast and lightweight. In my opinion it used to be more lightweight and now seems to have a lot of things being stuffed into it. I prefer to strip most of the extraneous elements I don't need and run with a minimal version of the web server that does just what I need - no more, no less. This is both more efficient, less bugs and more secure, since every line of code is a potential bug, takes CPU and memory to process, and is a potential security problem as well. This is personal preference - some people prefer bloated code which makes it easier and faster to get things done at the expense of the other factors.

Jetty was originally created by Greg Wilkins and team but has moved over to Codehaus and can be downloaded here:

Download Jetty

Jetty is used by some well known big technology companies (like Google).

A handy way to develop web apps using Jetty is to run Jetty within Eclipse using Maven. To do that configure the Jetty Plugin and configure it. I have a separate post about that if you're interested as I ran into some issues. Then you can right click on your pom.xml file and run Jetty to test and debug your application within Eclipse as you are working.

Jetty Plugin with Maven and Eclipse

Tomcat

I've also worked at a lot of large and small companies that use Tomcat as their Java web server. Tomcat supports the various Java components you might want to run and is a very good option:

Tomcat

I've used various other Java web servers and app servers such as JBoss, Resin, JRun and JServe [which I don't think even exists anymore] but I would stick with Tomcat or Jetty.

Spring

Spring is a framework for software development used by many companies. It has various components that can be used to configure and build software projects more quickly. It is built using various Software Design Patterns - a concept any good software developer will want to know and use.

Spring can be downloaded here:

SpringSource.org

Hibernate

Hibernate is a software framework used for accessing databases. Hibernate is commonly used along with Spring. It handles various important aspects of database connections that can lead to performance problems and can speed up development of software that works with databases.

Hibernate

Databases

A lot of Java applications end up using databases. The most common are MySQL, SQL Server and Oracle.

MySQL

MySQL has a free, open source version and is used for a lot of different types of projects, small and large. The enterprise version of MySQL is competitive with SQL Server in many ways. In general it's a database that has been much, much cheaper than SQL Server and probably one of the main reasons it has been heavily used besides being a decent solution for most projects - and open source if you're really that geeky and think you want to change database code. I've used MySQL at various companies and it worked just fine.

MySQL

SQL Server

Microsoft SQL Server has a lot of great tools and support and personally I've just used it a lot and like it. I know what it can do and get things done quickly so I typically stick with it given the choice. For large companies it has some features that help integrate with other Windows networking options and is beneficial for highly scalable and robust solutions that require replication, among other things. I also know that a while back Microsoft snagged some of the best database gurus from some of their competitors. I figure if the brightest and best are developing this product it's a good product to choose. It has not let me down. There is a SQL Server express version which appears to be free - check the license for what you can and cannot do with it - and a developer version which last time I checked was under $50. I've also heard of some hosting companies offering SQL Server for web clients at $30/month which is a pretty good deal. The managed companies I've talked to typically charge in the $200-300/month range for SQL Server. I think the limitation of the web version is mainly replication but you'll have to explore that further yourself.

Search for SQL Server downloads at Microsoft.com

Oracle

Some large companies use Oracle but coming from the Northwest in the shadow of the Mothership, I don't see Oracle so much around here anymore. I know it's used for financial applications and such but the web projects I've worked on recently haven't involved Oracle. Oracle used to be "the" database when I started out and was what I started working with...now any of these three options will most certainly meet the needs of the majority of projects.

Find out more about Oracle at Oracle

Access

Access is a good database for a single user on a local machine - say an accountant that needs to import data and run customized reports. Access is *not* a good solution for a web site, a multi-user or networked application. Choose one of the others.

There are other database options, but those are by far the most commonly used databases in business applications.