Skip to content

Welcome

Welcome to my blog about the software tools / techniques I have used.

The first posts, were originally hosted in a variety of places,  &  I thought it would be nice to bring them all together.   Its mainly a journal of  some of the things I have worked on, that might be useful to me in the future.  If it helps anyone else, then great.

I have been developing Web Applications with Domino / Lotus Notes since the 90s, mainly in areas where workflow & security are important. Before that I worked on financial systems with Foxpro, Oracle & PL/1.

In parallel with Domino, recently I have focused more on C# .NET, Sharepoint, JQuery , Javascript, CSS  & HTML5 .

Richard

Advertisements

Populate angular search filter using javascript & cookies

If AngularJS $cookies doesn’t work, the following code can be used to push Javascript created cookies into the angularjs world.  The following example is a people finder based on firstname & surname.    JQuery focusout() sets cookies to the values for First & Last name, with an expiry time of 30 minutes.  If you navigate back to this url in that 30 minutes,  the angularEG.js will trigger & prefill the search fields with the last used value.

HTML Config in form

< script src ="~/Scripts/angularEG.js" >< /script >
<input type="text" ng-model="search.Last_Name" id="lastname" class="search-query" ng-model-options="{debounce: 500}" >

<input type="text" ng-model="search.First_Name" id="firstname" class="search-query" ng-model-options="{debounce: 500}" >

JQuery Code in form:

$(function () {
 function setCookie(cname, cvalue, exdays) {
 var d = new Date();
var exp = (exdays * 24 * 60 * 60 * 1000) + 1800000;
 d.setTime(d.getTime() + (exp));
 var expires = "expires=" + d.toUTCString();
 document.cookie = cname + "=" + cvalue + ";" + expires + ";path=/";
}
$("#lastname").focusout(function () {
 setCookie('cookie_lastname', $('#lastname').val(), 0);
 });
$("#firstname").focusout(function () {
 setCookie('cookie_firstname', $('#firstname').val(), 0);
 });

 

Then in Scripts/angularEG.js

myApp.controller('dataController', function ($scope, $http) {
.....
var fcookie = getCookieVal("cookie_firstname");
var lcookie = getCookieVal("cookie_lastname");
..other code..
 $scope.search = { Last_Name: lcookie, First_Name: fcookie };
...other code.
function getCookieVal(name) {
//get the cookies
 var nameEQ = name + "=";
 var ca = document.cookie.split(';');
 for (var i = 0; i < ca.length; i++) {
 var c = ca[i];
 while (c.charAt(0) == ' ') c = c.substring(1, c.length);
 if (c.indexOf(nameEQ) == 0) return c.substring(nameEQ.length, c.length);
 }
 return "";
 }
....
})

 

 

 

 

Post SOAPUI XML to WebAPI

I wanted to write a SOAP consumer for SOAP UI XML using WebAPI2.0, because WCF wasn’t playing nicely.  I needed  to process SOAP in the following format (in this case from SOAPUI):

<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:tem="http://tempuri.org/">
  <soapenv:Header>
    <wsse:Security soapenv:mustUnderstand="0" xmlns:wsse="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-secext-1.0.xsd" xmlns:wsu="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-utility-1.0.xsd">
    <wsse:UsernameToken wsu:Id="UsernameToken-93E8DFC2DBD6AF2C5115005561877993">
    <wsse:Username>test1</wsse:Username>
    <wsse:Password Type="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-username-token-profile-1.0#PasswordText">passwordtext</wsse:Password>
    <wsse:Nonce EncodingType="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-soap-message-security-1.0#Base64Binary">nWFUtDng50qtbALaZ+/Q4w==</wsse:Nonce>
    <wsu:Created>2017-07-20T13:09:47.799Z</wsu:Created>
   </wsse:UsernameToken>
  </wsse:Security>
 </soapenv:Header>
 <soapenv:Body>
  <tem:uploadData>  
   <Field1>gero et</Field1>  
   <Field2>sonoras imperio</Field2>  
   <Field3>quae divum incedo</Field3>  
   <Field4>verrantque per auras</Field4>
  </tem:uploadData>
 </soapenv:Body>
</soapenv:Envelope>

Notice this has a Wsse Basic Auth credential in the header, & a simple XML Body.

Step 1 I built a standard WebApi2.0 project  (mine was called SoapService).

Step 2 I created a C# class for uploadData (the XML Node within the Body above).  My first attempt at this (with a more complex data model) used the raw SOAPUI XML with Visual Studio’s  Edit / Paste Special / Paste XML as Classes. 9 times out of 10 this works brilliantly. Today, the deserialize only parsed half the data I needed, & left many of the body’s  sub classes as null, giving no error message.  Instead,  I manually built a  simpler class matching the precise node,  I wanted to work on (uploadData):

[XmlRoot("uploadData")]
 public class uploadData
 {
  [XmlElement("Field1")] 
  public string Field1 { get; set; }
  [XmlElement("Field2")] 
  public string Field2 { get; set; }
  [XmlElement("Field3")] 
  public string Field3 { get; set; }
  [XmlElement("Field4")] 
  public string Field4 { get; set; }
}

Step 3 The basic POST function of the WebAPi’s SoapServiceController  takes in a string & echoes it back to you. I tweaked it as follows:

  1. Changed the input type from string to HttpRequest message, so you can get everything that is being posted, headers & all.
  2. Got the actual posted data with value.Content.ReadAsStringAsync().Result
  3. Loaded that to an XMLDocument, to manipulate the content programmatically
  4. Got the security header into a XMLNodeList with .GetElementsByTagName();
  5. Parsed the security header elements into some other elements.
  6. Loaded the uploadData XML into another XMLNodeList & built a c# uploadData object  from it.
//1
public string Post(HttpRequestMessage value)
 { //2
  var posteddata = value.Content.ReadAsStringAsync().Result;
  XmlDocument xmldoc = new XmlDocument();
  xmldoc.LoadXml(posteddata);//3
//4
  XmlNodeList UserNameToken = xmldoc.GetElementsByTagName("wsse:UsernameToken");
  string uid = "";
  string pw="";
  string nonce="";
  XmlNamespaceManager ns = new XmlNamespaceManager(xmldoc.NameTable);
  ns.AddNamespace("wsse", "http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-secext-1.0.xsd");
  foreach (XmlNode item in UserNameToken)
  {//5
   uid = item.SelectSingleNode("wsse:Username",ns).InnerText;
   pw = item.SelectSingleNode("wsse:Password",ns).InnerText;
   nonce = item.SelectSingleNode("wsse:Nonce",ns).InnerText;
  }
 XmlNodeList elemList = xmldoc.GetElementsByTagName("tem:uploadData");
//6
 string f1 = "";
 string f2="";
//etc.

foreach (XmlNode P in elemList)
 {
  f1 = P.SelectSingleNode("Field1").InnerText;
  f2 = P.SelectSingleNode("Field2").InnerText;
.........
 }
 uploadData UD = new uploadData {
  Field1= f1,
  Field2= f2,
 .......
 };
=DoStuffwithUploadedData(UD);
return "success";

}

Simple Kerberos Feature Detect in JQuery

Simple jquery $.ajax on a remote service.  In the restful web API  AppAuthorize is a custom .NET Filter that takes in a role as a parameter & returns True or False as to whether the current user is a member of that role.  It extends the WEB API 2.0 Authorise functionality from System.Web.Http.AuthorizeAttribute.

The AuthChk function is Authorised for people with the “READERS” role

GenericLibrary.SQLHandler is a custom class, comprising

  •  public int returncode { get; set; }
  • public string Message { get; set; }
[Route("api/myApp/AuthChk")]
 [HttpGet]
 [AppAuthorize("READERS")] 
 public GenericLibrary.SQLHandler AuthChk( string callback = "")
 {
 
 GenericLibrary.SQLHandler retval = new GenericLibrary.SQLHandler();
try
 {// Is there a valid credential
var usercred = System.Web.HttpContext.Current.User.Identity.Name;
 if (usercred != null)
 {
  retval.returncode = 0;
  retval.Message = "API Authentication = " + usercred;
 }
 else
 {
  retval.returncode = -1;
  retval.Message = "Error accessing credential in API";
 }
}
 catch (Exception ex)
 {
 retval.returncode = -1;
 retval.Message = "API Credential Error : " + ex.Message; 
 }
 return retval;
 }

Then  in the document.ready function

var KrbResultCheck = "Error";
 $(document).ready(function () {
 var KerbCheckurl = "api/myApp/AuthChk";
toastr.options = {.....  config  toastr settings
 }
 try{
 $.ajax({
 url: KerbCheckurl,
 contentType: 'application/json; charset=utf-8',
 type: 'GET',
 async: false,
 dataType: 'json',
 success: function (data) {
// If the service works then overwrite KrbResultCheck
 KrbResultCheck = data;
},
 error: function (xhr, status, error) { 
 var err = JSON.parse(xhr.responseText);
 toastr.error(" Web Service Authentication Check failed" + err.Message , "ERROR");
 }
 });
 }
 catch (exception)
 {
 toastr.error(" Another Exception " + exception, "ERROR");

}

if (KrbResultCheck == "Error")
 {
 toastr.error("<br/><br/><br/> Remote Authentication Check executed but did not return correct credential", "ERROR");
 }

 }

Getting all the records from a large SharePoint list with CSOM

I have often had problems getting all the rows from a large Sharepoint list.  Admin limits on the number of rows you can return often cause queries to fail.

The fix for this is to make your query return fewer records using a Where clause in your Caml query.  These can be problematic too as they have eccentric syntax, you don’t always know what data values are actually present in the list, what are the internal fieldnames, or whether the query still fall foul of the Admin limits?

The SharePoint ID field can help.  It’s an internal field that increments automatically to give you a row id.  Its always there, has a different value for each row, & its a simple number – so its perfect for splitting your list into manageable chunks.

To process an entire list, I looked for records whose id was between 1 & 1000, & added them to  a C# list I created in memory (&whose structure matched the SharePoint list).  Then I moved onto to the next chunk and so forth until I was done.

To my knowledge Sharepoint doesnt make available the minimum & maximum list values,  so how did I know when I was done?  You’ll recall I started with id range 1 – 1000, then 1001-2000, Chunk 3 was 2001 to 3000.   If a query returned no records , I incremented a counter of queries that had returned nothing ( called zerocount) by 1.   When this counter reached the arbitrary figure of 700,  I exited the loop.

This helped because in my test the ID field didnt start at 1 (I think it was 38,000o or something like that. This helped us move through the empty ranges until we find ranges that do return records. 

The code below starts with a startkey of 1 & an endkey of 1000 & looks for records whose ids fall in that range.   When its done,  it increments both key values by 1000 ( to give 1001 – 2000) and repeats.  To stop a permanent loop, we stop processing after n  queries have returned nothing, since the last one did actually return data.  To do this, I increment a counter if no data is returned.  Otherwise, I reset the counter to zero.  At some point this counter will reach a point, when I decide there’s probably nothing else to process – so I stop the program.  I set this figure to be 700 – that’s 700 queries & 700,000 ID values searched for and not found in SharePoint, before I decide that there’s probably no more data!  The number of records to process at a time & the number of queries with no rows you accept , is for you to decide.

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
using Microsoft.SharePoint.Client;
using System.Net;
using System.Configuration;
using System.Data;
namespace myNameSpace
{
    class Program
    {
//get the base url from app.config      
        static string baseurl = ConfigurationManager.AppSettings["baseURL"];
        static void Main(string[] args)
        {
//Get your start and end id values for the first chunk from app.config 
//In this case start = 1 & end = 1000
            string eval1 = ConfigurationManager.AppSettings["StartKey"];
            string eval2 = ConfigurationManager.AppSettings["EndKey"];
// Keep track of how many queries return no records.
            int zerocount = 0;
            ClientContext ctx = new ClientContext(baseurl);
            List spList = ctx.Web.Lists.GetByTitle("myList");
            clientContext.Load(spList);
            ctx.ExecuteQuery();
//C# list to match the structure of the SharePoint list & to store the data in memory 
            List<MyClass> reslist = new List<MyClass>();
           string caml = "";
           int counter = 0;       
           int ItemCnt = spList.ItemCount;
           if (spList != null && spList.ItemCount > 0)
           {               
               do
               {
// Query where the ID is between the start & end values 
                   CamlQuery camlQuery = new CamlQuery();
                   caml = string.Format(@"<View><Query><Where><And><Geq>
<FieldRef Name='ID' /><Value Type='Counter'>{0}</Value></Geq>
<Leq><FieldRef Name='ID' /><Value Type='Counter'>{1}</Value></Leq></And>
</Where></Query><ViewFields><FieldRef Name='ID' />
<FieldRef Name='myfield1' /><FieldRef Name='myfield2' />
</ViewFields><RowLimit>5000</RowLimit></View>", eval1, eval2);
                   camlQuery.ViewXml = caml;
                   ListItemCollection listItems = spList.GetItems(camlQuery);
                   clientContext.Load(listItems);
                   clientContext.ExecuteQuery();
                   if (listItems.Count == 0)
                   {// If we get no records increment the zerocount parm
                       zerocount++;
                       Console.WriteLine("Range "+eval1 +" - " +eval2 +" no records.  Zero records _ Chunk: " +  zerocount);                   
                   }
                   else
                   {
// Records found 
//Reset zerocount & add each returned row to the C# list of type<MyClass>
                       zerocount = 0;
                       foreach (ListItem it in listItems)
                       {  counter++;                           
                              if (it.FieldValues["myfield1"] != null)
                           { reslist.Add(new MyClass
                               {
// for this example Myclass is comprised 2 fields - myfield1 & myfield2
                                  myfield1= it.FieldValues["myfield1"].toString(),
                                  myfield2= it.FieldValues["myfield2"].oString()
                               });
                           }
                       }
                    }
// Finished processing that chunk. Now increment start & end values
// of the id range to process by 1000
                   eval1 = increment(eval1, 1000);
                   eval2 = increment(eval2, 1000); 
               } while (zerocount < 700);
// I chose to accept 700 consecutive queries where no data is 
// returned before abandoning the loop 

//  Do other stuff with the list          
               }
        }

        public static string increment(string val1, int val2)
        { int tval1 = Convert.ToInt32(val1);
            tval1 += val2;
            return tval1.ToString();
        }    
    }
}

These are the app settings from the app config:

<appSettings>
    <add key="baseUrl" value="http://mySPurl/" />
    <add key="StartKey" value="1" />
    <add key="EndKey" value="1000" />
</appSettings>

& myClass looks like

class myClass
 {
    public string myfield1{ get; set; }
    public string myfield2 { get; set; }
  }

IF I find code to get the maximum & minimum ids I’ll post it here

 

C# Automated Testing Complex object from Web Service

When your  Assert.AreEqual or CollectionAsser.AreEqual fails inspite of both objects containing the same columns & values, you’ll find a lot of complicated advice about overriding the underlying .Equals code for your classes.

This is a much simpler approach which works with any 1 dimensional object containing many fields.  The code tests a restful web service with many parms

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;

using Microsoft.VisualStudio.TestTools.UnitTesting;
using System.Net.Http;
using System.Net;

using Newtonsoft.Json;
using System.Reflection;

using myWebServiceApp.Models;
using myWebServiceApp.Controllers;
namespace myWebServiceApp.Controllers.Tests
{
[TestClass()]
public class myWebServiceAppTests
{
string urlbase = "http://localhost:37781/api/myService/function";

[TestMethod()]
public void Test1_Eligible()
{
//Set url to web service
 string url= urlbase+ "?Parm1=A&Parm2=B&Dob=1994-07-02";
// Set object to contain expected results
 myServiceResultsClass DesiredResults = createDesiredResults("1","0","SuccessResult");
//set object to contain actual results from web service
 myServiceResultsClass testresults = runurl(url);
//pass both to ReflectiveEquals function to see if both contain the same columns & values
 bool testequal = ReflectiveEquals(DesiredResults, testresults);
//Pass the result of testequal into Assert
 Assert.AreEqual(testequal, true, "CheckTest1_FoundandEligible Error");
}

[TestMethod()]
public void Test2()
{//Another Test
string url = urlbase + "?Parm1=FAILVAL&Parm2=B&Dob=1994-07-02";
myServiceResultsClass DesiredResults = createDesiredResults("0", "0", "ExpectedFailResult");
myServiceResultsClass testresults = runurl(url);
bool testequal = ReflectiveEquals(DesiredResults, testresults);
Assert.AreEqual(testequal, true, "NinoTest2_NotFound Error");
}


// Private function calls
private myServiceResultsClass runurl(string url)
{//Run web service & parse results into myServiceResultsClass type object
 myServiceResultsClass retval = new myServiceResultsClass();
 using (var wb = new WebClient()){
  string tmpretval = wb.DownloadString(url);
  retval = JsonConvert.DeserializeObject<myServiceResultsClass>(tmpretval);
 }
return retval;
}

private myServiceResultsClass createDesiredResults(string elStat, string errCod, string desc)
{//Code to build expected results object
myServiceResultsClass retval = new myServiceResultsClass();
retval=( new myServiceResultsClass(){EligibilityStatus=elStat,ErrorCode=errCod,Description=desc});
return retval;
}

private bool ReflectiveEquals(object first, object second)
{//Compare two objects that I expect to contain the same column headers & values
// Test they both have values
if (first == null && second == null)
{
 return true;
}
if (first == null || second == null)
{
 return false;
}
Type firstType = first.GetType();
if (second.GetType() != firstType)
{
 return false; // Or throw an exception
}
// Using 1st object as template, compare its columns & values with the equivalent in the 2nd
foreach (PropertyInfo propertyInfo in firstType.GetProperties())
{
 if (propertyInfo.CanRead)
 {
  object firstValue = propertyInfo.GetValue(first, null);
  object secondValue = propertyInfo.GetValue(second, null);
  if (!object.Equals(firstValue, secondValue))
   {
   return false;
   }
  }
 }
return true;
}
}
}

Simple Console App to generate 60k people records as JSON, based on Game of Thrones or Star Wars names

 

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
using System.Configuration;
using System.Data;
using System.Data.SqlClient;
using Newtonsoft.Json;

namespace GeneratePeople
{
 class Program
 {
 static void Main(string[] args)
 {
 List<JSONEmployee> emplist = new List<JSONEmployee>();
for (int i = 1; i <= 60000; i++)
 {
 JSONEmployee person = new JSONEmployee();
 emplist.Add(person);
 person = new JSONEmployee();
 }
 var sortedList = emplist.OrderBy(o => o.Last_Name).ToList();
 var JSONFILE = JsonConvert.SerializeObject(sortedList);
System.IO.File.WriteAllText(@"C:\myFolder\EmplistGOT.json", JSONFILE.ToString());
 }
 }

public class JSONEmployee
 {
 // string surnames = "Ackbar|Ameda|Amidala|Antillies|Anujo|Baba|Bandon|Bane|Beezer|Bibble|Binks|Binks|Brandes|Bridger|Calrissian|Childsen|Crumb|Daala|Dameron|Dan|Darklighter|Dofine|Dooku|Dowmeia|Drallig|Dukal|Dulin|DunLa|Eckener|Etima|Evazan|Fett|Fortuna|Ghintee|Gonn|Goren|Gunray|Haako|Hilse|Hudorra|Jace|Jade|Jeng|Joanson|Kee|Keeg|Kenobi|Koon|Lars|Letah|Lobot|Loneozner|Maac|Malak|Malreaux|Mateil|Maul|Melan|Minx|Mothma|Nadon|Nass|Nunb|Okand|Onassi|Ordo|Organa|Porkins|Qiyn|Racine|Ralter|Ren|Retrac|Rue|Schous|Scrambas|Shé|Shif|Sidious|Sing|Sivrak|Skywalker|Smadis|Solo|Stahn|Tarkin|Thrawn|Tonnika|Torsyn|Trooper|Tyranus|Vader|Vander|Vao|Vida|Vimdim|Windu|Zapalo|Mundi|Kryze|Madine|Plagueis|Rebo";
 
// string firstnames = "Ariq|Arleil|Aurra|Ayy|Bail|Bairdon|Baniss|Bib|Biggs|Boba|Bom|Borsk|Boss|Cad|Camie|Canderous|Cariso|Carth|Cellheim|Chad|Cin|Cornelius|Count|Crix|Dack|Dansra|Darth|Daultay|Del|Delva|Diskret|Elyhek|Ezra|Figrin|Graf|Han|Hat|Hela|Hol|Hugo|Jabba|Jango|JarJar|Jek|Jon|Kai|Karoly|Khaat|Ki-adi|Klatuu|Knol|Koth|Koyi|Kylo|Labria|Lak|Lando|Lufta|Luke|Lumas|Lumpawarrump|Lunae|Mace|Mara|Mas|Max|Meeko|Mission|Momaw|Mon|Nien|Nute|Obi-Wan|Owen|Padmé|Pello|Plo|Poe|Ponda|Qui-Gon|Rune|Ryle|Salacious|Sarrissa|Satine|Sayar|Scout|Sel|Senni|Shada|Shann|Shayl|Sheltay|Shmi|Sio|Tarados|Tas|Tundra|Wedge|Whie|Willhuff";

 string surnames = "Baelish|Baratheon|Bolton|Clegane|Drogo|Giantsbane|Greyjoy|Lannister|Mormont|Naharis|Seaworth|Snow|Sparrow|Stark|Targaryen|Tarly|Tyrell";
 string firstnames = "Arya|Brienne of|Bronn|Catelyn|Cersei|Daario|Daenerys|Davos|Ellaria|Gendry|Gilly|High|Jaime|Jaqen|Jeor|Joffrey|Jon|Jorah|Khal|Margaery|Missandei|Ned|Petyr|Ramsay|Robb|Robert|Roose|Samwell|Sandor|Sansa|Shae|Stannis|Talisa|Tyrion|Tywin";

 public string Employee_Number { get; set; }
 public string Last_Name { get; set; }
 public string First_Name { get; set; }
 public string User_ID { get; set; }
 public Random rno { get; set; }
 
 public JSONEmployee()
 {
 this.rno = new Random();
 this.First_Name = this.RandomElement(firstnames);
 this.Last_Name = this.RandomElement(surnames);
 this.Employee_Number = this.RandomNumber(1000, 9999999).ToString();
 var x = this.RandomNumber(1, 99).ToString();
 this.User_ID = this.First_Name.Substring(0, 1) + this.Last_Name.Substring(0, 3) + x ;
}

public string RandomElement(string rawdata)
 {
 string[] arraydata = rawdata.Split('|');
 Random r = new Random(Guid.NewGuid().GetHashCode());
 int element = r.Next(0, arraydata.Length - 1);
 return arraydata[element];
 }

public int RandomNumber(int from, int to)
 {
  Random r = new Random(Guid.NewGuid().GetHashCode());
  int retval = r.Next(from, to);
  return retval;
 }
}
}

Realtek HD Audio Voices Quiet Windows 7

Quite or underwater vocals on a stereo laptop, without any 5.1 audio

Another possible fix for this issue

 

1) In Control Panel, go to SOUND, not REALTEK HD AUDIO MANAGER

2) Double click the current set of speakers or click properties

3) Click the Enhancements tab

4) Ensure VOICE CANCELLATION, & PITCH SHIFT are  NOT checked

5) Ensure DISABLE ALL SOUND EFFECTS, & IMMEDIATE MODE ARE checked